We were curious about what methods we can use to manage large files that must be copied between sites with WAN-type latency and also restrict ourselves to processes available on the CA Identity Suite virtual appliance / Symantec IGA solution.
Leveraging VMware Workstation’s ability to introduce network latency between images, allows for a validation of a global password reset solution.
If we experience deployment challenges with native copy operations, we need to ensure we have alternatives to address any out-of-sync data.
The embedded CA Directory maintains the data tier in separate binary files, using a software router to join the data tier into a virtual directory. This allows for scalability and growth to accommodate the largest of sites.
We focused on the provisioning directory (IMPD) as our likely candidate for re-syncing.
Test Conditions:
- To ensure the data was being securely copied, we kept the requirement for SSH sessions between two (2) different nodes of a cluster.
- We introduce latency with VMware Workstation NIC for one of the nodes.
3. The four (4) IMPD Data DSAs were resized to 2500 MB each (a similar size we have seen in production at many sites).
4. We removed data and the folder structure from the receiving node to avoid any checksum restart processes from gaining an unfair advantage.
5. If the process allowed for exclusions, we did take advantage of this feature.
6. The feature/process/commands must be available on the vApp to the ‘config’ or ‘dsa’ userIDs.
7. The reference host/node that is being pulled, has the CA Directory Data DSAs offline (dxserver stop all) to prevent ongoing changes to the files during the copy operation.
Observations:
SCP without Compression: Unable to exclude other files (*.tx,*.dp, UserStore) – This process took over 12 minutes to copy 10,250 MB of data
SCP with Compression: Unable to exclude other files (*.tx,*.dp, UserStore) – This process still took over 12 minutes to copy 10,250 MB of data
Rsync without compression: This process can exclude files/folders and has built-in checksum features (to allow a restart of a file if the connection is broken) and works over SSH as well. If the folder was not deleted prior, then this process would give artificial high-speed results. This process was able to exclude the UserStore DSA files and the transaction files (*.dp & *.tx) that are not required to be copied for use on a remote server. Only 10,000 MB (4 x 2500 MB) was copied instead of an extra 250 MB.
Rsync with compression: This process can exclude files/folders and has built-in checksum features (to allow a restart of a file if the connection is broken) and works over SSH as well. This process was the winner, and; extremely amazing performance over the other processes.
Total Time: 1 min 10 seconds for 10,000 MB of data over a WAN latency of 70 ms (140 ms R/T)
Now that we have found our winner, we need to do a few post steps to use the copied files. CA Directory, to maintain uniqueness between peer members of the multi-write (MW) group, have a unique name for the data folder and the data file. On the CA Identity Suite / Symantec IGA Virtual Appliance, pseudo nomenclature is used with two (2) digits.
The next step is to rename the folder and the files. Since the vApp is locked down for installing other tools that may be available for rename operations, we utilized the find and mv command with a regular xpression process to assist with these two (2) steps.
Complete Process Summarized with Validation
The below process was written within the default shell of ‘dsa’ userID ‘csh’. If the shell is changed to ‘bash’; update accordingly.
The below process also utilized a SSH RSA private/public key process that was previously generated for the ‘dsa’ user ID. If you are using the vApp, change the userID to config; and su – dsa to complete the necessary steps. You may need to add a copy operation between dsa & config userIDs.
[dsa@pwdha03 ~/data]$ dxserver status
ca-prov-srv-03-impd-main started
ca-prov-srv-03-impd-notify started
ca-prov-srv-03-impd-co started
ca-prov-srv-03-impd-inc started
ca-prov-srv-03-imps-router started
[dsa@pwdha03 ~/data]$ dxserver stop all > & /dev/null
[dsa@pwdha03 ~/data]$ du -hs
9.4G .
[dsa@pwdha03 ~/data]$ eval `ssh-agent` && ssh-add
Agent pid 5395
Enter passphrase for /opt/CA/Directory/dxserver/.ssh/id_rsa:
Identity added: /opt/CA/Directory/dxserver/.ssh/id_rsa (/opt/CA/Directory/dxserver/.ssh/id_rsa)
[dsa@pwdha03 ~/data]$ rm -rf *
[dsa@pwdha03 ~/data]$ du -hs
4.0K .
[dsa@pwdha03 ~/data]$ time rsync --progress -e 'ssh -ax' -avz --exclude "User*" --exclude "*.dp" --exclude "*.tx" dsa@192.168.242.135:./data/ $DXHOME/data
FIPS mode initialized
receiving incremental file list
./
ca-prov-srv-01-impd-co/
ca-prov-srv-01-impd-co/ca-prov-srv-01-impd-co.db
2500000000 100% 143.33MB/s 0:00:16 (xfer#1, to-check=3/9)
ca-prov-srv-01-impd-inc/
ca-prov-srv-01-impd-inc/ca-prov-srv-01-impd-inc.db
2500000000 100% 153.50MB/s 0:00:15 (xfer#2, to-check=2/9)
ca-prov-srv-01-impd-main/
ca-prov-srv-01-impd-main/ca-prov-srv-01-impd-main.db
2500000000 100% 132.17MB/s 0:00:18 (xfer#3, to-check=1/9)
ca-prov-srv-01-impd-notify/
ca-prov-srv-01-impd-notify/ca-prov-srv-01-impd-notify.db
2500000000 100% 130.91MB/s 0:00:18 (xfer#4, to-check=0/9)
sent 137 bytes received 9810722 bytes 139161.12 bytes/sec
total size is 10000000000 speedup is 1019.28
27.237u 5.696s 1:09.43 47.4% 0+0k 128+19531264io 2pf+0w
[dsa@pwdha03 ~/data]$ ls
ca-prov-srv-01-impd-co ca-prov-srv-01-impd-inc ca-prov-srv-01-impd-main ca-prov-srv-01-impd-notify
[dsa@pwdha03 ~/data]$ find $DXHOME/data/ -mindepth 1 -type d -exec bash -c 'mv $0 ${0/01/03}' {} \; > & /dev/null
[dsa@pwdha03 ~/data]$ ls
ca-prov-srv-03-impd-co ca-prov-srv-03-impd-inc ca-prov-srv-03-impd-main ca-prov-srv-03-impd-notify
[dsa@pwdha03 ~/data]$ find $DXHOME/data -depth -name '*.db' -exec bash -c 'mv $0 ${0/01/03}' {} \; > & /dev/null
[dsa@pwdha03 ~/data]$ dxserver start all
Starting all dxservers
ca-prov-srv-03-impd-main starting
..
ca-prov-srv-03-impd-main started
ca-prov-srv-03-impd-notify starting
..
ca-prov-srv-03-impd-notify started
ca-prov-srv-03-impd-co starting
..
ca-prov-srv-03-impd-co started
ca-prov-srv-03-impd-inc starting
..
ca-prov-srv-03-impd-inc started
ca-prov-srv-03-imps-router starting
..
ca-prov-srv-03-imps-router started
[dsa@pwdha03 ~/data]$ du -hs
9.4G .
[dsa@pwdha03 ~/data]$
Note: An enhancement has been open to request that the ‘dsa’ userID is able to use remote SSH processes to address any challenges if the Data IMPD DSAs need to be copied or retained for backup processes.
Example for vApp Patches:
Note: There is no major different in speed if the files being copied are already compressed. The below image shows that initial copy is at the rate of the network w/ latency. The value gain from using rsync is still the checksum feature that allow auto-restart where it left off.
vApp Patch process refined to a few lines (to three nodes of a cluster deployment)
# PATCHES # On Local vApp [as config userID] mkdir -p patches && cd patches curl -L -O ftp://ftp.ca.com/pub/CAIdentitySuiteVA/cumulative-patches/14.3.0/CP-VA-140300-0002.tar.gpg curl -L -O ftp://ftp.ca.com/pub/CAIdentitySuiteVA/cumulative-patches/14.3.0/CP-IMV-140300-0001.tgz.gpg screen [will open a new bash shell ] patch_vapp CP-VA-140300-0002.tar.gpg [Patch VA prior to any solution patch] patch_vapp CP-IMV-140300-0001.tgz.gpg exit [exit screen] cd .. # Push from one host to another via scp IP=192.168.242.136;scp -r patches config@$IP: IP=192.168.242.137;scp -r patches config@$IP: # Push from one host to another via rsync over ssh [Minor gain for compressed files] IP=192.168.242.136;rsync --progress -e 'ssh -ax' -avz $HOME/patches config@$IP: IP=192.168.242.137;rsync --progress -e 'ssh -ax' -avz $HOME/patches config@$IP: # Pull from one host to another via rsync over ssh [Minor gain for compressed files] IP=192.168.242.135;rsync --progress -e 'ssh -ax' -avz config@$IP:./patches $HOME # View the files were patched IP=192.168.242.136;ssh -tt config@$IP "ls -lart patches" IP=192.168.242.137;ssh -tt config@$IP "ls -lart patches" # On Remote vApp Node #2 IP=192.168.242.136;ssh $IP cd patches screen [will open a new bash shell ] patch_vapp CP-VA-140300-0002.tar.gpg patch_vapp CP-IMV-140300-0001.tgz.gpg exit [exit screen] exit [exit to original host] # On Remote vApp Node #3 IP=192.168.242.137;ssh $IP cd patches screen [will open a new bash shell ] patch_vapp CP-VA-140300-0002.tar.gpg patch_vapp CP-IMV-140300-0001.tgz.gpg exit [exit screen] exit [exit to original host]
View of rotating the SSH RSA key for CONFIG User ID
# CONFIG - On local vApp host ls -lart .ssh [view any prior files] echo y | ssh-keygen -b 4096 -N Password01 -C $USER -f $HOME/.ssh/id_rsa IP=192.168.242.135;ssh-keyscan -p 22 $IP >> .ssh/known_hosts IP=192.168.242.136;ssh-keyscan -p 22 $IP >> .ssh/known_hosts IP=192.168.242.137;ssh-keyscan -p 22 $IP >> .ssh/known_hosts cp -r -p .ssh/id_rsa.pub .ssh/authorized_keys rm -rf /tmp/*.$USER.ssh-keys.tar tar -cvf /tmp/`/bin/date -u +%s`.$USER.ssh-keys.tar .ssh ls -lart /tmp/*.$USER.ssh-keys.tar eval `ssh-agent` && ssh-add [Enter Password for SSH RSA Private Key] IP=192.168.242.136;scp `ls /tmp/*.$USER.ssh-keys.tar` config@$IP: IP=192.168.242.137;scp `ls /tmp/*.$USER.ssh-keys.tar` config@$IP: USER=config;ssh -tt $USER@192.168.242.136 "tar -xvf *.$USER.ssh-keys.tar" USER=config;ssh -tt $USER@192.168.242.137 "tar -xvf *.$USER.ssh-keys.tar" IP=192.168.242.136;ssh $IP `/bin/date -u +%s` IP=192.168.242.137;ssh $IP `/bin/date -u +%s` IP=192.168.242.136;ssh -vv $IP [Use -vv to troubleshoot ssh process] IP=192.168.242.137;ssh -vv $IP [Use -vv to troubleshoot ssh process]
