Is Copy-n-Paste operations impacting your Identity & Governance solutions?

Microsoft Office Suite’s Autocorrect: How Character Replacements Impact Identity and Governance Solutions => Garbage-In-Garbage-Out (GIGO)

When thinking about identity and governance solutions, many of us consider factors such as password security, multi-factor authentication, or access control. Rarely do we contemplate the subtle implications of character replacements in our word processing software. However, Microsoft Office Suite’s Autocorrect feature, while intended to enhance the user experience, has introduced concerns around the copy-paste process, especially with characters like the dash and quotes. Let’s delve into the nuances of this issue and its potential impacts for two (2) of the most common replacements that have impact.

A Common Scenario:

Automated emails from Ticket Systems are forwarded to administrators or users, then these admin/users may copy-n-paste these strings from the email (or MS word document) to an identity / governance solution, as they wish to be efficient and ensure no mistyped characters happen from one solution to another. These fields could be used for provisioning access by a business role name or kicking off a governance campaign search.

Dash vs. Emdash: What’s the Big Deal?

Microsoft Word (and other programs within the Office Suite) has a habit of automatically converting the standard dash (-) to an emdash (—) when it assumes the user is attempting to create a longer break in the sentence. On the surface, this appears to be a simple formatting choice. Yet, when you copy content containing these characters and paste them into identity or governance platforms, unexpected issues may arise. This “emdash” decision appears to be following British style formatting per this reference. https://www.sussex.ac.uk/informatics/punctuation/hyphenanddash/dash

Identity systems often depend on exact character matching for elements like usernames, role names, domain names, or system strings. For instance, if a user is instructed to input “domain-name.com” but inadvertently pastes “domain—name.com” (with an emdash), the system will not recognize the latter as a valid entry. This leads to failed authentication attempts, locked accounts, and potential security concerns as users and admins scramble to correct the discrepancies. Worst case, the identity/governance solution is using UTF-8 or newer character sets to accept the special characters, but the underlying IG/IM database is still using older ASCII format, that do not recognize the newer character sets. If this occurs, then a data clean up operation is typically needed by the IM/IG/DBA teams.

The Smart Quotes Dilemma

Similarly, Microsoft’s Autocorrect feature replaces standard double quotes (“) with smart quotes (“ ”) for a more visually appealing look in documents. While they may enhance the aesthetic feel of a document, smart quotes can wreak havoc in systems expecting the simpler ASCII version.

A code or script that depends on specific string matching will fail if smart quotes are used instead of standard quotes. This can lead to malfunctioning applications, scripts, or integrations when developers or administrators copy and paste content from Office documents directly into configuration files or codebases.

Governance Solutions and Data Integrity

In governance solutions, consistency and data integrity are of the utmost importance. Consider a scenario where policy documents or terms of use agreements are drafted in Word. Any auto-replaced characters might be unintentionally added to official records or database entries. When such documents are parsed or processed by automated systems, unexpected behaviors might occur due to these seemingly innocuous character changes.

Recommendations and Best Practices:

  1. Awareness: Ensure that your team is aware of these auto-corrections. Training sessions or instructional guides can be used to inform users about these pitfalls.
  2. Disable Autocorrect: If you frequently copy and paste between Office Suite and other platforms, consider disabling these specific autocorrect features for these two (2) common ones (dash/quotes). See the below screen shots how to disable these two (2) features in MS Outlook, MS Word, and MS Powerpoint. Fortunately, we do not have to modify MS Excel. From a global updates, companies may wish to visit their patch process, to update the MS registry for these auto correction behavior for all users.
  3. Post-Copy Verification: After pasting content, always double-check critical characters to ensure they have not been auto-replaced. It may be necessary to incorporate policy verification rules to prevent entry of these two (2) common replacement characters, e.g. PX Policy UI data verification rules.
  4. Use Plain Text Editors: When dealing with sensitive or system-related information, use plain text editors like Notepad, Notepad++ or VSCode to avoid any auto-formatting.

Location of auto-correction of dash (-) to emdash (–) & quotes in MS Outlook

Location of auto-correction of dash (-) to emdash (–) & quotes in MS Word

Location of auto-correction of dash (-) to emdash (–) & quotes in MS Powerpoint

Fortunately, we do NOT have this issue in MS Excel for the two (2) characters we are reviewing in this blog.

An impact of copy-n-paste:

For example, if you are using an Oracle database, and you may see upside down question mark characters ¿ in your data sets, this is a strong indicator that the database is doing an auto-replacement for the special characters that it does not recognize. The below example showcases when users/administrators would use copy-n-paste operations to create new IM/IG objects, that would not be returned when searching later upon these objects, as the names would no longer match what was entered the 1st time.

If the database has a default character map, this effort will not be simple, as the DBAs must make a major change and will require an outage window. The DBAs may also need to be involved in the data clean up or replacement exercise to adjust the malformed entries.

Conclusion

The Microsoft Office Suite’s Autocorrect feature demonstrates how even well-intentioned, user-friendly functionalities can introduce unforeseen challenges. For those operating in the realm of identity and governance, an awareness of these issues is essential. It’s a testament to the intricate nature of modern software environments, where even the simplest character can have significant implications. Confirm your identity access / governance solutions have a matching character set between the solution stack and the underlying database.

Leveling Up: The Imperative of Upgrading Your Symantec Identity Suite Virtual Appliance to 14.5 (Centos Stream 9) for Robust Randomness, Enhanced Jitterentropy, and Bouncy Castle Entropy Insights

In the intricate world of cybersecurity and identity management, evolving threats and vulnerabilities demand our undivided attention. When considering upgrading your Symantec Identity Suite Virtual Appliance, understanding the nuanced technological landscape, including the perks of Jitterentropy and the challenges associated with Java’s Bouncy Castle entropy, can make a world of difference.

The Technological Need:

  1. Robust Randomness with Jitterentropy: Relying on the natural timing jitter of CPUs, Jitterentropy has emerged as a game-changing hardware random number generator (RNG). The latest renditions of the Symantec Identity Suite Virtual Appliance leverage this RNG, ensuring unparalleled randomness, making decoding by potential threats a herculean task.
  2. Operational Efficiency: Upgrades tuned with contemporary features promise optimized performance. Coupled with Jitterentropy, the RNG processes are turbocharged, promising minimal downtime and an elevated user experience.
  3. Challenges with Bouncy Castle Entropy in Java: Bouncy Castle, despite its vast utility in cryptographic operations in Java, has had its share of entropy-related issues. Some known problems include:
  • Predictability: Certain RNG implementations in Bouncy Castle have been found to be somewhat predictable, which could compromise security.
  • Seed Reuse: There have been instances where seeds were reused, which again poses security concerns.
  • Slow Entropy Accumulation: At times, the entropy collection is slower than expected, leading to potential operational delays. With security solutions the lack of entropy impacts scale and usability.

Business Justification for Rapid Response:

With the business landscape in perpetual flux, the right tech decisions can spell the difference between stagnation and growth:

  1. Enhanced Security: Incorporating Linux OS with Jitterentropy is synonymous with state-of-the-art security. Such forward-thinking measures drastically curtail potential security breaches.
  2. Cost Savings: Forward-looking upgrades, especially those that incorporate cutting-edge features like Jitterentropy, offer tangible long-term financial advantages. Fewer breaches, reduced system errors, and saved manual efforts contribute positively to the bottom line.
  3. Staying Competitive: In an era of rapid technological advancements, integrating elements like Jitterentropy ensures you’re leading from the front.
  4. Compliance and Regulatory Adherence: With cybersecurity standards constantly on the move, staying updated is non-negotiable. Evade potential legal issues and hefty fines by staying on top of these norms.
  5. Customer Trust: By showcasing a commitment to data safety through advanced systems (and by addressing known entropy issues like those in Bouncy Castle), businesses can strengthen customer trust and foster long-term loyalty.

Validating Jitterentropy Integration in the Linux Kernel: A Comprehensive Guide

As the world of Linux continues to evolve, one exciting development is the incorporation of jitterentropy into the kernel. This robust hardware random number generator (RNG) enhances the quality of randomness, making our systems even more secure. If you’re keen on understanding, implementing, or validating this feature in your Linux setup, this guide is tailored just for you.

What is Jitterentropy?

Jitterentropy is an RNG based on the natural timing jitter that occurs in CPUs. In the realm of cybersecurity, RNGs are of paramount importance; they generate the random numbers pivotal for cryptographic operations. The less predictable these numbers are, the tougher it becomes for malicious actors to crack them.

Why is Jitterentropy Essential?

For systems relying on cryptographic functions, such as encryption, the RNG’s caliber can’t be overstated. Jitterentropy guarantees first-rate randomness, upping your system’s security game. https://www.chronox.de/jent.html

How to Validate Jitterentropy Integration:

  1. Identify Your Kernel Version:
    Kick things off by determining your kernel version using the uname -r or uname -acommand.
   uname -r

This will provide insights into your system’s hostname, kernel version, build date, and architecture. You can deterermine if your Linux kernel is greater than 5.6, when entropy functionality was added directly to the kernel. https://github.com/torvalds/linux/commit/3f2dc2798b81531fd93a3b9b7c39da47ec689e55

  1. Is Jitterentropy Part of Your Kernel Configuration?:
    Deploy this simple grep command to figure out if jitterentropy is enabled in your kernel:
   grep -HRin jitter /boot/config*

An output showing CONFIG_CRYPTO_JITTERENTROPY=y confirms that jitterentropy is enabled. The “y” here indicates that the feature is in-built in the kernel.

  1. Time-Driven Testing for Jitterentropy:
    By simulating multiple pulls from the entropy source, you can gauge how efficient jitterentropy is:
   time for i in {1..1000}; do time dd if=/dev/random bs=1 count=16 2>/dev/null | base64; done

This command performs two functions:

  • It times each of the 1000 pulls from /dev/random, allowing you to measure the average time taken, basically emulating 1000 rapid password changes of 16 characters.
  • It provides an overall timing for 1000 pulls, letting you know the total duration for the entire operation. If your system remains responsive and completes the pulls swiftly, it’s a strong indication that your entropy source is in prime working condition. Which implies that any solution on the appliance has adequate entropy to service users and processes to scale.

Another command that add counters to see that 1000 iteration have passed. Note, if there is no entropy pump, this process will NOT succeed. The Linux OS entropy will be rapidly depleted and any solution on the host will be delayed. Ensure there is an entropy pump to keep the performance you need.

counter=1;MAX=1000;time while [ $counter -le $MAX ]; do echo "##########  $counter ##########" ; time dd if=/dev/random bs=16 count=1 2> /dev/null | base64; counter=$(( $counter + 1 )); done;

Wrapping Up:

The integration of Jitterentropy in the Linux kernel underscores the open-source community’s relentless dedication to fortifying security. By understanding, testing, and leveraging it, you ensure that your system is bolstered against potential threats, always staying a step ahead in the cybersecurity arena. Keep exploring, stay updated, and most importantly, remain secure!

Review upgrade your Symantec Identity Suite to improve your performance for users and scale to millions of transactions.

For non-appliances or older Linux OS (Kernel release < 5.6):

Review adding the haveged or jitterentropy packages to your Linux OS, to avoid delays to any business processes. See prior blog discussing entropy, of how adding an entropy pump to your Linux OSes has value. https://anapartner.com/2021/06/25/the-hidden-cost-of-entropy-to-your-business/

VMware Workstation and Vyos Software Router: Expedite on-prem Kubernetes and OpenShift Labs

With the rapid evolution of technology and increasing complexity of software solutions, using tools like VMware Workstation for learning and testing has become necessary. Deploying intricate systems like Kubernetes and OpenShift on VMware Workstation provides an opportunity for in-depth understanding and experience before implementing these solutions on a larger, organization-wide scale.

VMware Workstation, coupled with the powerful container orchestration capabilities of Kubernetes and OpenShift, offers an unparalleled platform for crafting next-generation applications and solutions and lowering costs. It’s a potent combination that can significantly boost your organization’s operational efficiency, application delivery speed, and overall software development lifecycle.

In the realm of advanced solution deployments, the right tools can make all the difference. With VMware Workstation, you’re not just getting a virtualization tool; you’re acquiring a platform that helps you delve deeper into modern software architectures and innovations. Harness its potential and equip yourself with the knowledge and experience needed to stay ahead of the curve.

Certainly, networking is one of the critical aspects of VMware Workstation that make it such a versatile tool. VMware Workstation offers three types of networking options to suit different needs and scenarios. Let’s explore each of these in detail.

1. Bridged Networking

Bridged Networking is the simplest and most straightforward networking mode. When you configure a VM to use bridged networking, the VM is connected directly to the existing network that your host computer is connected to. In essence, it will be as though the VM is another physical device on your network.

With bridged networking, your VM can have its unique identity on the network, such as its IP address, making it an entirely independent entity from the host. This is particularly useful when you need the VM to interact directly with other devices on the network, or when it needs to be accessible from other computers.

2. Network Address Translation (NAT)

The NAT mode allows your VMs to share the IP address of the host machine. Essentially, all the network traffic from the VMs is routed through the host machine. This implies that the VMs can access the external network and the internet, but they cannot be directly reached from the external network since they are ‘hidden’ behind the host.

NAT is highly beneficial when you want to isolate your VMs from your network while still providing them with network access. For instance, this can be handy when testing untrusted applications or experimenting with potentially unstable software that could disrupt your network.

3. Host-Only Networking

The Host-Only networking mode creates a private network shared only between the VMs and the host machine. This means that your VMs can communicate with each other and the host machine but cannot access the external network or the internet.

Host-Only networking is particularly useful when you want to create a secure, isolated environment for your VMs, away from the vulnerabilities of the external network. This is ideal when working with sensitive data or creating a controlled environment for testing network applications.

Each of these three VMware Workstation networking modes has advantages and suitable use-cases. The choice between them depends on your specific needs- creating an isolated testing environment or mimicking a complex, interconnected network for a comprehensive deployment simulation.

Expanding Host-Only for use with OpenShift/Kubernetes Labs

As discussed earlier, VMware workstation offer three (3) types of networks modes: Bridged, NAT, and Host Only. The bridged mode has a challenge that it will share your office or home network and request an IP address to be assigned. This may not be acceptable in your office, or you may wish to keep your main home network free from VMware hosts. NAT is typically the most selected network used for VMware guest OS, as it will not impact the office/home network. The limitation with NAT, is it only allows outward-bound traffic from the Guest OS, via the VMware Host. There are no routing rules to allow traffic from outside to access the Guest OS images. The last network mode is Host-Only. Host-Only is designed to be an isolated network segment between the VMware guest OS and the VMware Host OS. There is no outward or inward-bound traffic. This network mode is typically not used when access to the internet is required.

Introduce: Vyos Software Router for VMware (OVA)

We wanted a more flexible solution than these three (3) modes. We wanted to standardize a network segment for our OpenShift/Kubernetes training/development that did not require a change between locations (like bridged) or force our internal resources to reset their bridged network to match.

After a review, we selected VMware Host-Only, which has the basics of what we needed. We were only missing routing rules for inbound and outbound traffic. We looked around and found a software solution already made that we could immediately leverage with minimal configuration changes to Vmware client OS/images. Vyos software router was already provided in an OVA format for immediate use.

We downloaded and imported the OVA into VMware workstation.

Since we planned to have multiple host network segments to manage large data for OpenShift/Kubernetes, we bumped up the VMware guest OS specs from 1 vCPU 4 GB RAM to 2 vCPU 8 GB RAM. And adjusted the extra Network Adapters to be Host-only or Custom (Host-Only) networks.

After we adjusted the Guest OS specs, we snapshotted this VMware Guest OS image to allow rollback if we wanted to change a feature later. We started up the image and logged in with default credentials; vyos/vyos

After login via the VMware Guest OS console, we immediately updated Vyos configuration to allow us to ssh into the Guest OS and perform our work in a better UI.

Below is an example of the bootstrap configuration to enable remote access via ssh, and update eth0 NIC to a bridged IP address that we can access. We standardized a rule that all network routing would use IP xxx.yyy.zzz.254.

conf
set service ssh port '22'
set interfaces ethernet eth0 address '192.168.2.254/24'
commit
save

We then switched to our favorite SSH terminal tool of MobaXterm (or Putty) to validate we could access the Vyos software router remotely.

We are now ready to add a configuration that allows a default route, inbound routes, and outbound routes for our four (4) network NICs.

The below lines may be pasted into the SSH session. ‘conf’ (config) will open the Vyos configuration shell so that we can paste it into all lines. We will define static IP addresses for all four (4) NICs, a static route to our external network router, outbound rules, and inbound rules. Please ensure that the IP addresses for the four (4) NICs match what you have defined.

conf
set service ssh port '22'

set interfaces ethernet eth0 address '192.168.2.254/24'
set interfaces ethernet eth0 description 'BRIDGED NETWORK'

set interfaces ethernet eth1 address '10.10.10.254/24'
set interfaces ethernet eth1 description 'VMWARE HOST NETWORK vmnet1'

set interfaces ethernet eth2 address '10.0.0.254/24'
set interfaces ethernet eth2 description 'VMWARE HOST NETWORK vmnet2 - BAREMETAL OPENSHIFT'

set interfaces ethernet eth3 address '192.168.242.254/24'
set interfaces ethernet eth3 description 'VMWARE HOST NETWORK vmnet3'

delete protocols static route 0.0.0.0/0
set protocols static route 0.0.0.0/0 next-hop 192.168.2.1

delete nat

set nat source rule 20 description "Allow Outbound Traffic from VMware Host network from eth1"
set nat source rule 20 outbound-interface 'eth0'
set nat source rule 20 source address '10.10.10.0/24'
set nat source rule 20 translation address masquerade

set nat source rule 30 description "Allow Outbound Traffic from VMware Host network from eth2"
set nat source rule 30 outbound-interface 'eth0'
set nat source rule 30 source address '10.0.0.0/24'
set nat source rule 30 translation address masquerade

set nat source rule 40 description "Allow Outbound Traffic from VMware Host network from eth3"
set nat source rule 40 outbound-interface 'eth0'
set nat source rule 40 source address '192.168.242.0/24'
set nat source rule 40 translation address masquerade

set nat source rule 60 description "Allow Inbound Traffic from Bridged to VMware host network eth1"
set nat source rule 60 outbound-interface 'eth1'
set nat source rule 60 source address '192.168.2.0/24'
set nat source rule 60 translation address masquerade

set nat source rule 61 description "Allow Inbound Traffic from Bridged to VMware Host network eth2"
set nat source rule 61 outbound-interface 'eth2'
set nat source rule 61 source address '192.168.2.0/24'
set nat source rule 61 translation address masquerade

set nat source rule 62 description "Allow Inbound Traffic from Bridged to Vmware Host network eth3"
set nat source rule 62 outbound-interface 'eth3'
set nat source rule 62 source address '192.168.2.0/24'
set nat source rule 62 translation address masquerade

commit
save
exit
show interface
show ip route 0.0.0.0

Please double check the IP addresses match your VMware Host-only networks.

Validation

We will validate inbound and outbound traffic using ping on the Vyos software router. When this passes, we will move on to routing configuration for external devices.

After basic validation, please snapshot your Vyos Guest OS

In the final step, we will add routing configuration on MS Windows OS and Linux OS to reach all four (4) networks from any external device and any VMware image on one of the four (4) networks.

# Ref: https://docs.vyos.io/en/equuleus/configuration/system/default-route.html
#      https://docs.vyos.io/en/equuleus/quick-start.html
#      https://bertvv.github.io/cheat-sheets/VyOS.html

#Step 000:  Increase Vyos Router specs from 1 vCPU 4 GB RAM to 2 vCPU 8 GB RAM when adding more than two interfaces in VMware Workstation


#Step 00: Review VMware Host vmnet addresses, use to build your rules.

ip a | grep vmnet

16: vmnet1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UNKNOWN group default qlen 1000
    inet 10.10.10.1/24 brd 10.10.10.255 scope global vmnet1
17: vmnet2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UNKNOWN group default qlen 1000
    inet 10.0.0.1/24 brd 10.0.0.255 scope global vmnet2
18: vmnet3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UNKNOWN group default qlen 1000
    inet 192.168.242.1/24 brd 192.168.242.255 scope global vmnet3
19: vmnet8: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UNKNOWN group default qlen 1000
    inet 192.168.243.1/24 brd 192.168.243.255 scope global vmnet8
20: vmnet255: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UNKNOWN group default qlen 1000
    inet 10.255.0.1/24 brd 10.255.0.255 scope global vmnet255



# Step 0:  Boot strap first interface (via vmware console of vyos running image -  after login with vyos / vyos)

conf
set service ssh port '22'
set interfaces ethernet eth0 address '192.168.2.254/24'
commit
save
exit
show interface


# Step 1: Vyos configuration - after login with vyos / vyos with an SSH putty session tool to allow copy-n-paste of the below rows

conf
set service ssh port '22'

set interfaces ethernet eth0 address '192.168.2.254/24'
set interfaces ethernet eth0 description 'BRIDGED NETWORK'

set interfaces ethernet eth1 address '10.10.10.254/24'
set interfaces ethernet eth1 description 'VMWARE HOST NETWORK vmnet1'

set interfaces ethernet eth2 address '10.0.0.254/24'
set interfaces ethernet eth2 description 'VMWARE HOST NETWORK vmnet2 - BAREMETAL OPENSHIFT'

set interfaces ethernet eth3 address '192.168.242.254/24'
set interfaces ethernet eth3 description 'VMWARE HOST NETWORK vmnet3'

delete protocols static route 0.0.0.0/0
set protocols static route 0.0.0.0/0 next-hop 192.168.2.1

delete nat

set nat source rule 20 description "Allow Outbound Traffic from VMware Host network from eth1"
set nat source rule 20 outbound-interface 'eth0'
set nat source rule 20 source address '10.10.10.0/24'
set nat source rule 20 translation address masquerade

set nat source rule 30 description "Allow Outbound Traffic from VMware Host network from eth2"
set nat source rule 30 outbound-interface 'eth0'
set nat source rule 30 source address '10.0.0.0/24'
set nat source rule 30 translation address masquerade

set nat source rule 40 description "Allow Outbound Traffic from VMware Host network from eth3"
set nat source rule 40 outbound-interface 'eth0'
set nat source rule 40 source address '192.168.242.0/24'
set nat source rule 40 translation address masquerade

set nat source rule 60 description "Allow Inbound Traffic from Bridged to VMware host network eth1"
set nat source rule 60 outbound-interface 'eth1'
set nat source rule 60 source address '192.168.2.0/24'
set nat source rule 60 translation address masquerade

set nat source rule 61 description "Allow Inbound Traffic from Bridged to VMware Host network eth2"
set nat source rule 61 outbound-interface 'eth2'
set nat source rule 61 source address '192.168.2.0/24'
set nat source rule 61 translation address masquerade

set nat source rule 62 description "Allow Inbound Traffic from Bridged to Vmware Host network eth3"
set nat source rule 62 outbound-interface 'eth3'
set nat source rule 62 source address '192.168.2.0/24'
set nat source rule 62 translation address masquerade

commit
save
exit
show interface
show ip route 0.0.0.0 


# Step 2:  Update external lab network devices (laptop on 192.168.2.x) to use Vyos Router for this new routes

# MS Win OS examples:
route add -p 10.10.10.0 mask 255.255.255.0 192.168.2.254
route add -p 10.0.0.0 mask 255.255.255.0 192.168.2.254
route add -p 192.168.242.0 mask 255.255.255.0 192.168.2.254

ping 10.10.10.254
ping 10.0.0.254
ping 192.168.242.254

# Linux OS examples:
sudo route add -net  10.0.0.0/24 gw 192.168.2.254
sudo route add -net  10.10.10.0/24 gw 192.168.2.254
sudo route add -net  192.168.242.0/24 gw 192.168.2.254
route -n
netstat -rn   (dnf -y install net-tools)

ping 10.10.10.254
ping 10.0.0.254
ping 192.168.242.254

# Step 3:  Optional:  Add static routes on network router if missed on a device, to redirect to the vyos bridged interface.


# Step 4:  Update the VMware DHCP configuration file to use the new Vyos Router for any Vmware images with DHCP, then reboot images.
option routers  10.10.10.254;     [VMware Workstation on Linux OS: /etc/vmware/vmnet1/dhcp/dhcpd.conf ]
option routers  10.0.0.254;       [VMware Workstation on Linux OS: /etc/vmware/vmnet2/dhcp/dhcpd.conf ]
option routers  192.168.242.254;  [VMware Workstation on Linux OS: /etc/vmware/vmnet3/dhcp/dhcpd.conf ]

# Note:  MS Win OS:  The VMware DHCP configurations are combined in one file:  C:\ProgramData\VMware\vmnetdhcp.conf
# 
# Restart images, view routes, then do a outbound submission as a test.


ping 8.8.8.8
ping www.google.com


# Step 5:  For Openshift, ensure that your install-config.yaml or agent-config.yaml is defined with the correct gateway router for Vyos.



# Step 6:  Exercise your VMware host images and then monitor within Vyos via:
show nat source translations
show nat source statistics
monitor traffic interface any filter 'host 10.0.0.99'      [embedded tcpdump]

Overview of Vyos Software Router with Vmware Workstation and three (3) host-only networks with bridged network

We now have the methodology to use over 250+ possible VMware host-only network segments for our networking labs with OpenShift and Kubernetes that require internet outbound and/or inbound access. We can standardize a unique host-only network segment to share with team members and clients for training/education/development. With the embedded tcpdump feature in Vyos Software router image, we can quickly address and isolate network routing configuration challenges.

Hopefully, this will allow you to continue to expand your knowledge and awareness of new architectures with your dedicated lab environment.

Secure Application Introspection

Locate “the good, the bad, and the ugly” data with a transparent proxy.

Have you been frustrated with various enterprise/cloud solutions’ APIs implementation or documentation where a single case-sensitive data field entry delays progress? Does the solution have undocumented features for older client tools? Do you wish to know what your mobile apps or laptop sends to the internet?

Utilizing a proxy can help with all the above, and if the process is quick and straightforward, so much the better.

Typically for a proxy, there may be quite a bit of effort and steps. You may need to modify a client host/mobile phone to redirect web traffic with OS environmental variables of HTTP_PROXY and HTTPS_PROXY or adjustment of the underlying OS network/iptables. Prior, we typically set up the open-source Jmeter proxy with the OS environment variables to capture secure traffic data. This process works well for most applications. Additionally, the Firefox browser allows manual modification using a proxy without dependence on the OS environment settings if we wish to capture the user experience and any data challenges.

The example below of modifying a Firefox browser to use a “manual proxy configuration” instead of system/auto configurations.

To ensure accurate capture of web traffic submissions, a more thorough method is needed as the above process may fail if client tools or mobile apps cannot detect OS environmental variables.

We have found a perfect combination within the open-source tool of MITMproxy with podman and the embedded VPN feature of WireGuard.

The process in six (6) steps:

  1. Deployment of the WireGuard VPN client on the client host (MS Win/Linux/Mobile)
  2. Deployment of MITMproxy using podman (or docker) with WireGuard mode/configuration
  3. Edit the wireguard.conf file to have the correct public IP address and import this file to the WireGuard VPN client and establish the VPN connection.
  4. Copy the mitmproxy-ca-cert.cer to the client component Java or OS keystore (if needed) as a trusted CA cert.
  5. Open the MITMproxy Web UI or monitor the command line dashboard
  6. Execute your test on the client host and view the results in the MITMproxy Web UI for both request and response.

MITMproxy UI with WireGuard mode enabled.

The WireGuard client configuration will be provided in three (3) places: the MITMproxy logs (podman logs mitmproxy), the text file wireguard.conf (if podman/docker volumes are enabled), and the MITMproxy UI. The QR code is enabled for mobile phone use, but since the public IP address provided is not correct in this view, you will need to manually edit this configuration on your mobile phone during those use-cases to have the correct endpoint IP address.

MITMproxy UI with standard proxy configuration mode.

Bash Script:

Script to deploy MITMproxy with podman on a linux OS with two (2) configurations: Wireguard mode for any client applications that do not honor HTTP_PROXY/HTTPS_PROXY and Standard proxy mode. This bash script allows a shared volume to use the SAME certs to avoid managing different certs upon restart of the container.

#!/bin/bash
######################################################################################
#
#  Deploy MITMproxy with two (2) configurations:
#
#     MITMProxy with WireGuard mode enabled (UDP 51820) and Web UI (TCP 8081)
#     MITMProxy with standard proxy enabled (TCP 9080) and Web UI (TCP 9081)
#
#  Notes:  Use podman exec to check path and env variables
#    - Binaries:  dnf -y install podman 
#    - Use shared folder to avoid having two (2) different configuration files for both copies
#    - Do not forget the :z for -v volumes to avoid permissions issues
#    - Do not forget quotes around env -e variables
#    - Use --rm as needed
#    - Use this switch as needed, but do not leave it on:   --log-level debug \
#
#   Basic:  podman run -it -v /tmp/mitmproxy/:/home/mitmproxy/.mitmproxy:z -p 8080:8080 mitmproxy/mitmproxy
#   Logs:   podman logs mitmproxy-wireguard
#   Shell:  podman exec -it -u root mitmproxy bash
#
#  Options Ref.  https://docs.mitmproxy.org/stable/concepts-options/
#   - added stream_large_bodies=10m to lower impact to mitmproxy due
#       to possible large json/xml payloads 
#
#  ANA 07/2023
#
######################################################################################

MITMPROXY_HOMEPATH=/tmp/mitmproxy
echo ""
echo "You may delete the shared folder of ${MITMPROXY_HOMEPATH}"
echo "to remove prior configuration of mitmproxy certs & wireguard.conf files"
echo ""
#sudo rm -rf ${MITMPROXY_HOMEPATH}

mkdir -p ${MITMPROXY_HOMEPATH}
chmod -R 777 ${MITMPROXY_HOMEPATH}
ls -hlrt ${MITMPROXY_HOMEPATH}

echo ""
echo " Starting mitmproxy-wireguard proxy "
podman rm mitmproxy-wireguard -f  &>/dev/null
podman run -d -it --name mitmproxy-wireguard \
   -p 51820:51820/udp -p 8081:8081 \
   -l mitmproxy \
   -v ${MITMPROXY_HOMEPATH}:/home/mitmproxy/.mitmproxy:z  \
    docker.io/mitmproxy/mitmproxy \
    mitmweb --mode wireguard --ssl-insecure  --web-host 0.0.0.0 --web-port 8081 --set stream_large_bodies=10m


echo ""
echo " Starting mitmproxy-default proxy "
podman rm mitmproxy-default -f  &>/dev/null
podman run -d -it --name mitmproxy-default \
    -p 9080:9080 -p 9081:9081 \
    -l mitmproxy  \
    -v ${MITMPROXY_HOMEPATH}:/home/mitmproxy/.mitmproxy:z  \
     docker.io/mitmproxy/mitmproxy \
     mitmweb --set listen_port=9080 --web-host 0.0.0.0 --web-port 9081

echo ""
echo ""
echo "###############################################################################"
echo ""
echo " Running Podman Containers for MITMproxy"
sleep 5
podman ps -a --no-trunc | grep -i mitmproxy
echo ""
echo "###############################################################################"
podman logs  mitmproxy-default
echo ""
echo " Monitor the mitmproxy-default UI @ http://$(curl -s ifconfig.me):9081 "
echo "###############################################################################"
podman logs  mitmproxy-wireguard
echo ""
echo " Monitor the mitmproxy-wireguard UI @ http://$(curl -s ifconfig.me):8081 "
echo "###############################################################################"
echo ""
echo "Please update the mitmproxy wireguard client configuration endpoint address to:  $(curl -s ifconfig.me)"
echo ""
echo "###############################################################################"
echo ""

MITMproxy CERTS:

Add mitmproxy-ca-cert to the trusted root certs folder on your client host OS keystore (MS Win: certlm.msc) and/or if there is a java keystore for the client tool, please add the mitmproxy-ca-cert.cer as a trusted cert. keytool -import -trustcacerts -file mitm-ca-proxy.cer -alias mitmproxy -keystore capam.keystore

WireGuard client configuration:

To ensure that only selected web traffic is monitored through wireguard VPN to mitmproxy, make changes to the wireguard.conf file before importing it. Specifically, update the AllowedIPs address field to include a single IP address. Additionally, modify the endpoint field to direct traffic to the public IP address of the mitmproxy host on UDP port 51820. If deploying mitmproxy on AWS or other cloud hosts, confirm that the firewall/security groups permit TCP 8080, 8081, 9080, 9091, and UDP 51820. Once you have activated the WireGuard client, test your processes on the host and monitor the MITMproxy UI for updates.

An example of data captured between two (2) CLI tools. These CLI tools did not honor the OS environmental variables of HTTP_PROXY & HTTPS_PROXY. Using the MITMproxy with WireGuard process, we can now confirm the delta submission behavior that was masked by the CLI tools. This process was useful to confirm that MS Powershell was removing special characters for a password string, e.g. ! (exclamation mark).

Example of script deploying two (2) MITMproxy containers

What changed? Business Logic XML Delta Process

When your solution grows, it may be challenging to identify when and what team members added with regards to new business logic. We can parse through change control documentation, but that may be a long and frustrating process. One of the challenges is that objects created within the Symantec (Broadcom) Identity Suite solution may not have date stamps on the object within the database tables. Again, we could parse through logs and the Task Persistence and Archive Task Persistence databases.

Please stop this behavior.

Let’s introduce a new streamlined process to help your administrative and business IAG teams.

Below is a process to use existing tools & samples within the existing IAG solution.

Goals: Automate a daily backup and create an XML delta of new business objects that were created the prior days. The tools used will be the included Import Export utility with additional Linux commands. We focused this process on the Symantec (Broadcom) Identity Suite Virtual Appliance with the built-in userID of ‘config’.

To start we will copy the solution’s included Import/Export sample to the ‘config’ userID home folder, and the minimal libraries files required to execute this process. We will then modify the script to perform a delta compare every day that it is executed by the ‘config’ crontab.

# Step 1 - Create a copy of the Import / Export Utility under the config home folder

mkdir -p /home/config/backup/export
cp -r -p /opt/CA/IdentityManager/IAM_Suite/IdentityManager/tools/ImportExportUtility/ /home/config/backup/export/


# Step 2 - Copy the three (3) extra JAR files required by the Import / Export Utility

mkdir -p /home/config/backup/export/lib
cp -r -p /opt/CA/IdentityManager/IAM_Suite/IdentityManager/tools/lib/idmutils.jar /home/config/backup/export/lib/
cp -r -p /opt/CA/IdentityManager/IAM_Suite/IdentityManager/tools/lib/log4j.jar /home/config/backup/export/lib/
cp -r -p /opt/CA/IdentityManager/IAM_Suite/IdentityManager/tools/lib/bc-fips.jar /home/config/backup/export/lib/

# Step 3 - Backup the current shell script and properties file before our changes.

cd /home/config/backup/export/ImportExportUtility/
cp -r -p config.properties config.properties.org
cp -r -p ImportExportUtil.sh ImportExportUtil.sh.org

We now will update the config.properties with your own hostname and credentials.

Below is the contexts of a working file, with additional commentary to assist with the replacement of the PBES password encryption. Please note, that the mode=export, and that we have selected resourceType=RoleDefination. Over 99% of the business objects will reside within this single XLM file when it is exported. We set the localPath=. to be the current path to allow the automated scripts to rename files for use with an XML diff tool. You may wish to update the export path to a network path.

## provide IM server base url with port number 
## Use netstat -apn | grep 8080 to confirm IP address
baseUrl=http://192.168.2.220:8080
## Login credential (in case Management console is protected), use JSAFE algorithm of PasswordTool to encrypt your plain text password
## Example: /opt/CA/IdentityManager/IAM_Suite/IdentityManager/tools/PasswordTool > ./pwdtools.sh -JSAFE -p Password01
userName=admin
password={PBES}:B8+4u/F3aiZ9sXus6HyDNA==
## provide mode import/export
mode=export
## provide resource type ALL/Directory/Environment/RoleDefinition
#resourceType=Directory
resourceType=RoleDefinition
#resourceType=Environment
#resourceType=All
## provide comma separated list of Directories to import/export, in case of import it should be xml file name
directories=ProvStore,UserStore,AuthenticationDirectory
## provide Environment name for Environment/Role Definition import/export, in case of environment import it should be zip file name
environment=identityEnv
## In case of Role Definition import please provide xml file name
roleDefFileName=env-RoleDefinitions
## provide local path to save/get the resources, in case of export directory structure will be created
localPath=.
## provide request time out in minutes
timeout=600
## restart Environment after import: yes/no, For restart to work environment name should be provided
restartEnv=no

We now will update the shell script of ImportExportUtil.sh.

We have renamed this shell script to ExportRoles.sh to clearly call out what we wish for this process to focus on, and what we will call it via a crontab entry. We have enhanced the JAVA_OPTS to speed up exports depending on the number of business objects, e.g. an IME with 40K provisioning roles may take over 60 minutes to export. We then have created a process that will generate an XML diff between the prior export and the latest export. We do date-time stamp the exports to allow past review of changes.

#!/bin/sh
#
#  This batch file sets up the environment and runs the IM Import and Export Utility
#

if [ -z "$JAVA_HOME" ] ; then
  echo "---------------------------------------------------------------------"
  echo "ERROR: Cannot find JAVA_HOME"
  echo "Please specify JAVA_HOME variable in this script file."
  echo "---------------------------------------------------------------------"
  exit
fi

export JAVA_HOME

MYCLASSPATH=.:./importExportUtility.jar:../lib/bc-fips.jar:../lib/idmutils.jar:../lib/log4j.jar

export MYCLASSPATH

##############
### ANA, 04/22
# Update JAVA_OPTS for speed as 40K Prov Roles may take 60-90 minutes to export
tz=`/bin/date --utc +%Y%m%d%H%M%S,3%N.0Z`
echo ""
echo "Starting at : $tz"
echo ""
#JAVA_OPTS="-Xms256m -Xmx512m $JAVA_OPTS"

JAVA_OPTS="-Xms256m -Xmx4g -Djava.net.preferIPv4Stack=true -Djava.awt.headless=true -Djava.security.egd=file:/dev/./urandom -Dcom.sun.jndi.ldap.object.disableEndpointIdentification=true $JAVA_OPTS"


##############
### ANA, 04/22
# Rename prior backup to be used for diff operation prior to exporting a new file
dos2unix identityEnv-RoleDefinitions.xml                                           >/dev/null 2>&1
cp -r -p identityEnv-RoleDefinitions.xml  identityEnv-RoleDefinitions_prior.xml    >/dev/null 2>&1

export JAVA_OPTS
$JAVA_HOME/bin/java $JAVA_OPTS -cp $MYCLASSPATH com.ca.identitymanager.importexportutility.client.ImportExportClient


##############
### ANA, 04/22
# Setup config crontab to execute this task every day at 1:11 AM
#  11 1 * * *   /home/config/scripts/create_pr_and_import_them/ExportRoles/ExportRoles.sh  >/dev/null 2>&1
# Use https://crontab.guru/ to define the correct scheduler
# Rename vApp (all) files for future review and delta compares

dos2unix identityEnv-RoleDefinitions.xml                                           >/dev/null 2>&1
cp -r -p identityEnv-RoleDefinitions.xml    identityEnv-RoleDefinitions_$tz.xml    >/dev/null 2>&1


##############
### ANA, 04/22
# Perform Diff operation between prior and new exports of IME Roles&Tasks.xml files
# Create an XML diff between two prior files.
#diff <(xmllint --c14n identityEnv-RoleDefinitions_prior.xml) <(xmllint --c14n identityEnv-RoleDefinitions.xml)

xmllint --c14n identityEnv-RoleDefinitions_prior.xml                                       > identityEnv-RoleDefinitions_prior_xmllint.xml
xmllint --c14n identityEnv-RoleDefinitions.xml                                             > identityEnv-RoleDefinitions_xmllint.xml
diff identityEnv-RoleDefinitions_prior_xmllint.xml identityEnv-RoleDefinitions_xmllint.xml > identityEnv-RoleDefinitions_DIFF_xmllint.xml
cp -r -p identityEnv-RoleDefinitions_DIFF_xmllint.xml identityEnv-RoleDefinitions_DIFF_xmllint_$tz.xml    >/dev/null 2>&1
echo ""
echo "There are `wc -l identityEnv-RoleDefinitions_DIFF_xmllint.xml` rows different between the two files"
echo ""
echo "There are `grep -i   '<ImsRole' identityEnv-RoleDefinitions_DIFF_xmllint.xml | wc -l ` Roles delta between the two files"
echo ""
echo "Please review if these deltas are correct:  cat identityEnv-RoleDefinitions_DIFF_xmllint.xml | more "
echo ""
tz=`/bin/date --utc +%Y%m%d%H%M%S,3%N.0Z`
echo "Done at : $tz "
echo  ""

Examples of this process executed. First, let’s generate about 5 provisioning roles to be loaded into the IME and then import them.

config@pwdha001 VAPP-14.3.0 (192.168.2.220):~/scripts/create_pr_and_import_them > ./create_5k_pr_xml_file_for_ime_mgmt_import.sh

real    0m0.028s
user    0m0.006s
sys     0m0.013s

Number of Provisioning Roles Created in XML File: 5
-rw-rw-r-- 1 config config 6.2K Apr 23 14:01 new_provisioning_roles.xml

Now we will import this XML file into the IME (see sample below for a Provisioning Role addition)

<?xml version="1.0" encoding="UTF-8"?>
<ims:ImsTemplate xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://imsenvironmentobjects/xsd imsconfig://schema/ImsEnv
ironmentObjects.xsd" xmlns:ims="http://imsenvironmentobjects/xsd" xmlns:imsrule="http://imsmemberrule/xsd" xmlns:imsscope="http://imsscoperule/xsd" x
mlns:imschange="http://imschangeaction/xsd">

        <!--   ******************** Create 10K Provisioning Roles ********************   -->

        <ImsRole name="prov-role_160823241129774" roletype="PROVISIONING" assignable="true" adminassignable="true" enabled="true" allowduplicatecusto
m="false" description="DESC_upto_128_Characters" custom01="CF01_upto_1000_Characters" custom02="CF02" custom03="CF03" custom04="CF04" custom05="CF05"
 custom06="CF06" custom07="CF07" custom08="CF08" custom09="CF09" custom10="CF10">

                <AdminPolicy assignable="true" adminassignable="true">
                        <imsrule:MemberRule><RoleMember><AdminRole name="User Manager"/></RoleMember></imsrule:MemberRule>
                        <imsscope:ScopeRule object="USER" purpose="*"><All/></imsscope:ScopeRule>
                </AdminPolicy>
                <AdminPolicy assignable="true" adminassignable="true">
                        <imsrule:MemberRule><All/></imsrule:MemberRule>
                        <imsscope:ScopeRule object="USER" purpose="*"><All/></imsscope:ScopeRule>
                </AdminPolicy>

                <OwnerPolicy>
                        <imsrule:MemberRule><RoleMember><AdminRole name="System Manager"/></RoleMember></imsrule:MemberRule>
                </OwnerPolicy>

                <Attribute name="comments">2022-04-23T20:01:47.000Z : COMMENTS_upto_128_Characters</Attribute>
                <Attribute name="department">DEPT_upto_100_Characters</Attribute>
     </ImsRole>

</ims:ImsTemplate>

Output from an import with the ImportExportUtil shell script

-----------------------------------------------------------
-------------------Starting a new Import-------------------
-----------------------------------------------------------
Importing Role Definition to Environment 'identityEnv'...
#############  Import Output  #############
Warning: Updating the IdentityMinder environment "identityEnv"
  Deploying role definitions...
    Importing Roles...

*********
0 error(s), 0 warning(s)
Role Definition Imported Successfully!!!

Now we will run our new ExportRoles.sh script, where it will export the IME and do an XML delta compare operation between a prior export and the latest export.

config@pwdha001 VAPP-14.3.0 (192.168.2.220):~/backup/export/ImportExportUtility > time ./ExportRoles.sh

Starting at : 20220423190707,3582824826.0Z

-----------------------------------------------------------
-------------------Starting a new Export-------------------
-----------------------------------------------------------
Exporting Role Definition from Environment 'identityEnv'...
 disposition attachment; filename=identityEnv-RoleDefinitions.xml;
Exported Filename: identityEnv-RoleDefinitions.xml
Role Definition exported Successfully!!!

There are 76 identityEnv-RoleDefinitions_DIFF_xmllint.xml rows different between the two files

There are 5 Roles delta between the two files

Please review if these deltas are correct:  cat identityEnv-RoleDefinitions_DIFF_xmllint.xml | more

Done at : 20220423190713,3241475139.0Z


real    0m5.663s
user    0m2.509s
sys     0m0.394s

Here is a view of the delta file that is generated. We now KNOW what was added since the last export. We have a date range as well.

>       <ImsRole adminassignable="true" allowduplicatecustom="false" assignable="true" custom01="CF01_upto_1000_Characters" custom02="CF02" custom03="CF03" custom04="CF04" custom05="CF0
5" custom06="CF06" custom07="CF07" custom08="CF08" custom09="CF09" custom10="CF10" description="DESC_upto_128_Characters" enabled="true" name="prov-role_160823241129774" roletype="PROVI
SIONING">
>               <AdminPolicy adminassignable="true" assignable="true">
>                       <imsrule:MemberRule><RoleMember><AdminRole name="User Manager"></AdminRole></RoleMember></imsrule:MemberRule>
>                       <imsscope:ScopeRule object="USER" purpose="*"><All></All></imsscope:ScopeRule>
>               </AdminPolicy>
>               <AdminPolicy adminassignable="true" assignable="true">
>                       <imsrule:MemberRule><All></All></imsrule:MemberRule>
>                       <imsscope:ScopeRule object="USER" purpose="*"><All></All></imsscope:ScopeRule>
>               </AdminPolicy>
>               <OwnerPolicy>
>                       <imsrule:MemberRule><RoleMember><AdminRole name="System Manager"></AdminRole></RoleMember></imsrule:MemberRule>
>               </OwnerPolicy>
>               <Attribute name="comments">2022-04-23T20:01:47.000Z : COMMENTS_upto_128_Characters</Attribute>
>               <Attribute name="department">DEPT_upto_100_Characters</Attribute>
>       </ImsRole>

A view of the files generated in the export process. NOTE: The first time executed, there will be an error due to the missing 1st file. Make a change in the IME, and then run this process a 2nd time to see the delta.

For larger IME, the export may take 60-90 minutes. The below example shows for a full “ALL” export that may be configured in the config.properties file. This IME was updated with over 40K Provisioning Roles to help isolate a java memory leak.

Counting the roles (Admin, Access, Provisioning) within the IME (Oracle DB) table of IM_ROLE

When we tested loading 5K Provisioning Roles, we noted it would take about 30 minutes. If the input file was over 10 MB, then the Import process would fail with a generic java error.

Lets Encrypt DNS Challenge

We are fond of the Let’s Encrypt DNS challenge process instead of alternative processes. The Let’s Encrypt DNS challenge using the certbot allows businesses to scale their replacements of certs that are not exposed directly to the internet. The certbot tool has switches that allow custom scripts to be run, which allows for a lot of flexibility.

Installation of certbot

Review the website for https://certbot.eff.org/, and there are sections per OS on how to use “snap” to install this tool. Example with CentOS7. https://certbot.eff.org/instructions?ws=other&os=centosrhel7

sudo snap install core; sudo snap refresh core
sudo snap install --classic certbot
sudo ln -s /snap/bin/certbot /usr/bin/certbot

We have used certbot with manual steps every 90 days with our development DNS domains but wanted to automate these steps. Unfortunately, we noticed a challenge with using Google Domains not having an API available to update DNS records. After research, we did find that Google Cloud does have the APIs available.

We had several options:

a) Move the DNS domains from Google Domains to Google Cloud,

b) redirect CNAME records from Google Domains to Google Cloud,

c) move to another Domain Register that has APIs available,

d) redirect CNAME records from Google Domain to another Domain Register.

Since we had another Domain register offering APIs, we decided to choose option d. This entry will review our steps and how we leverage certbot and the two (2) DNS Domains Registers.

Step 1: Google Domains – Create _acme-challenge CNAME records.

Step 2: 2nd Domain Register – GoDaddy – Create _acme-challenge TXT records.

Step 3: Enable the API Key on the 2nd Domain Register and the API Password Code

https://developer.godaddy.com/keys?hbi_code=1

Step 4: Validate via CURL the ability to update the TXT records through the 2nd Domain Register’s API.

curl -s -X PUT \
"https://api.godaddy.com/v1/domains/anapartner.in/records/TXT/_acme-challenge.aks.iam.anapartner.dev" \
-H  "accept: application/json" -H  "Content-Type: application/json" \
-H  "Authorization: sso-key ${GODADDY_API_KEY}:${GODADDY_API_SECRET}" \
-d "[{ \"data\": \"TESTING THIS STRING FIELD", \"name\": \"_acme-challenge.aks.iam.anapartner.dev", \"ttl\": 600 }]"

Finally, we will create a script that will be executed by crontab every 85 days. Please note that the scripts to be called by certbot are created as HERE DOCS to allow portability within a single script.

#!/bin/bash
###############################################################################
#
#  Update Google DNS via round-about way through 2nd DNS Register DNS API (anapartner.in)
#
#
#  Pre-work:
#     1. Use existing or purchase a domain from 2nd DNS register
#
#     2. Create Google Domain CNAME records for each of the wildcard domain to a remote DNS TXT Record
#     _acme-challenge.gke.iam.anapartner.org CNAME _acme-challenge.gke.iam.anapartner.org.anapartner.in
#     _acme-challenge.aks.iam.anapartner.org CNAME _acme-challenge.aks.iam.anapartner.org.anapartner.in
#     _acme-challenge.eks.iam.anapartner.org CNAME _acme-challenge.eks.iam.anapartner.org.anapartner.in
#
#     3. Create 2nd DNS Register Domain TXT records for each of the object to be updated
#     _acme-challenge.gke.iam.anapartner.org.baugher.net
#     _acme-challenge.aks.iam.anapartner.org.baugher.net
#     _acme-challenge.eks.iam.anapartner.org.baugher.net
#
#     4. Enable the 2nd DNS Register API for Production Access (Developer) & store the KEY & SECRET for use
#        https://developer.godaddy.com/keys?hbi_code=1
#
#     5. Install certbot     dnf -y install certbot
#         Note:  certbot will use two (2) variables of:  CERTBOT_DOMAIN (after the -d switch)
#         and  CERTBOT_VALIDATION (the text string to be used for TXT records)
#
#  ANA 07/2022
#
###############################################################################

GODADDY_API_KEY="XXXXXXXXXXXXXy"
GODADDY_API_SECRET="XXXXXXXXXXXXXXX"
DOMAIN="anapartner.in"


echo ""
echo "Create wildcard domain list"
echo "This may be any TXT record for a remote domain FQDN that is mapped in the anapartner.in"
echo "#####################################################################"

#cat << 'EOF' > wildcard-domains.txt
#*.gke.iam.anapartner.in
#*.aks.iam.anapartner.in
#*.eks.iam.anapartner.in
#EOF

cat << 'EOF' > wildcard-domains.txt
*.gke.iam.anapartner.dev
*.aks.iam.anapartner.dev
*.eks.iam.anapartner.dev
EOF

WILDCARD_DOMAIN=anapartner.dev

echo ""
echo "Create godaddy.sh script to update TXT records"
echo "#####################################################################"
cat <<  EOF > godaddy.sh
#!/bin/bash
if [[ "\$CERTBOT_DOMAIN" =~ .*anapartner.in* ]];then
    echo "If domain contains anapartner.in, we need to remove the last part to avoid duplicates during registration"
    CERTBOT_DOMAIN="\${CERTBOT_DOMAIN/".anapartner.in"//}"
    echo \$CERTBOT_DOMAIN
fi

DNS_REC_NAME="_acme-challenge.\$CERTBOT_DOMAIN"


curl -s -X PUT \
"https://api.godaddy.com/v1/domains/${DOMAIN}/records/TXT/\${DNS_REC_NAME}" \
-H  "accept: application/json" -H  "Content-Type: application/json" \
-H  "Authorization: sso-key ${GODADDY_API_KEY}:${GODADDY_API_SECRET}" \
-d "[{ \"data\": \"\$CERTBOT_VALIDATION\", \"name\": \"\${DNS_REC_NAME}\", \"ttl\": 600 }]"

sleep 30
EOF

chmod 555 godaddy.sh

echo ""
echo "Create godaddy-clean.sh script to wipe TXT records - as needed"
echo "#####################################################################"
cat << EOF > godaddy-clean.sh
#!/bin/bash

if [[ "\$CERTBOT_DOMAIN" =~ .*anapartner.in* ]];then
    echo "If domain contains anapartner.in, we need to remove the last part to avoid duplicates during registration"
    CERTBOT_DOMAIN="\${CERTBOT_DOMAIN/".anapartner.in"//}"
    echo \$CERTBOT_DOMAIN
fi

DNS_REC_NAME="_acme-challenge.\$CERTBOT_DOMAIN"

curl -s -X PUT \
"https://api.godaddy.com/v1/domains/${DOMAIN}/records/TXT/\${DNS_REC_NAME}" \
-H  "accept: application/json" -H  "Content-Type: application/json" \
-H  "Authorization: sso-key ${GODADDY_API_KEY}:${GODADDY_API_SECRET}" \
-d "[{ \"data\": \"clean\", \"name\": \"\${DNS_REC_NAME}\", \"ttl\": 600 }]"

EOF
chmod 555 godaddy-clean.sh


echo ""
echo "Start Loop to use Let's Encrypt's certbot tool"
echo "#####################################################################"
while read -r domain;
do

echo "#####################################################################"
echo "$domain"
echo ""
certbot -d $domain --agree-tos --register-unsafely-without-email --manual \
--preferred-challenges dns --manual-auth-hook ./godaddy.sh \
--manual-cleanup-hook ./godaddy-clean.sh --manual-public-ip-logging-ok \
--force-renewal certonly

echo ""

done < wildcard-domains.txt
# Add logic to handle the certs/keys when they are issued.

echo ""
echo "#####################################################################"
ls -lart /etc/letsencrypt/archive/*

#rm -rf godaddy.sh godaddy-clean.sh &>/dev/null
echo ""

echo ""
echo "After validation, the TXT records will be marked with the 'clean' string "
echo "#####################################################################"
echo "nslookup  -type=txt _acme-challenge.eks.iam.$WILDCARD_DOMAIN 8.8.8.8 | head -6"
echo "nslookup  -type=txt _acme-challenge.aks.iam.$WILDCARD_DOMAIN 8.8.8.8 | head -6"
echo "nslookup  -type=txt _acme-challenge.gke.iam.$WILDCARD_DOMAIN 8.8.8.8 | head -6"

View of the script being executed

Files generated by Let’s Encrypt certbot [certN.pem, privkeyN.pem, chainN.pem, and fullchainN.pem]

View the JMS HornetQ Queue

Typically, we may use various tools to view JMS queue(s) related metrics for trends and stale/stuck activity. During issues with J2EE JMS Queue, though, it would be helpful to be able to view and trace transactions to assist with a resolution. With proper logging levels enabled, Wildfly/JBOSS logs show detailed information containing the JMS IDs associated with each transaction. These JMS transactions we see in the logs are already ‘in-flight’ and are being processed by a message handler.

On the Symantec Identity Suite Virtual Appliance, the Wildfly & HornetQ processes are run under the ‘wildfly’ service ID. The wildfly journals are located in the wildfly data folder and stored in a format that is efficient for processing. To perform analysis on the data within these journals, though, we noticed a challenge with read-permissions for the HornetQ files even when Wildfly/Java process is not actively running.

To avoid this issue on the Virtual Appliance, copy the HornetQ files to a temporary folder. Remember to copy the entire folder, including sub-folders.

mkdir -p /tmp/hornetq; cd /tmp/hornetq

cp -r -p /opt/CA/wildfly-idm/standalone/data/live-hornetq ./

java -cp "/opt/CA/wildfly-idm/modules/system/layers/base/io/netty/main/*:/opt/CA/wildfly-idm/modules/system/layers/base/org/hornetq/main/*:/opt/CA/wildfly-idm/modules/system/layers/base/org/jboss/logging/main/*" org.hornetq.tools.Main print-data /tmp/hornetq/live-hornetq/bindings  /tmp/hornetq/live-hornetq/journal

Once the live-hornetq folder is available in a tmp location, execute the below process for printing Journal content.

Print HornetQ Journal and Bindings

To export the HornetQ Journal Files to XML, the Java module of “org.hornetq.core.journal.impl.ExportJournal” requires the journal sub-folder with the prefix of “hornetq-data”, the file extension (hq), the file sizes, and where to export the XML file (export.dat). The prefix and file extension (hq) are unique to the Identity Suite vApp.

mkdir -p /tmp/hornetq; cd /tmp/hornetq

cp -r -p /opt/CA/wildfly-idm/standalone/data/live-hornetq ./

java -cp "/opt/CA/wildfly-idm/modules/system/layers/base/io/netty/main/*:/opt/CA/wildfly-idm/modules/system/layers/base/org/hornetq/main/*:/opt/CA/wildfly-idm/modules/system/layers/base/org/jboss/logging/main/*" org.hornetq.core.journal.impl.ExportJournal  /tmp/hornetq/live-hornetq/journal hornetq-data hq  25485760  /tmp/hornetq/export.dat
Export HornetQ Journal

The body/rows of the JMS export is partially base64. You may parse through this information as you wish.

Use this information to trace through transactions in the JMS queue.

For Cleanup, within the Symantec Identity Suite vApp, there are a few options. The first is deleting the JMS queue journals before starting the Wildfly service. This can be accomplished using the build-in alias ‘deleteIDMJMSqueue’.

alias deleteIDMJMSqueue='sudo /opt/CA/VirtualAppliance/scripts/.firstrun/deleteIDMJMSqueue.sh'

Another option is to remove a select JMS entry from the queue using /opt/CA/wildfly-idm/bin/jboss-cli.sh process. If created with an input script, escape the colons in the GUID.

/subsystem=transactions/log-store=log-store/:probe()

ls /subsystem=transactions/log-store=log-store/transactions

/subsystem=transactions/log-store=log-store/transactions=0:ffffa409cc8a:1c01b1ff:5c7e95ac:eb:delete() 

View a description of the JMS Processing from Broadcom Engineering/Support Teams (see below video)

This write-up provides the tools required for a deeper analysis. Debugging issues with JMS may test one’s patience, stay the course, stay persistent, and have fun!

References: (Delete JMS queue and remove a single entry)

https://knowledge.broadcom.com/external/article/233003/inprogress-task-issues-a-clients-guide.html

https://knowledge.broadcom.com/external/article/129101/arjuna016037-could-not-find-new-xaresour.html

Kubernetes and Vmware Workstation

Kubernetes was designed for the deployment of applications to cloud architecture with containers. Another way of thinking about Kubernetes; it gets us “out-of-the-install-binaries” business and focuses our efforts on the business value of a solution. We have documented our process of how we train our resources and partners. This process will help your team to excel and gain confidence with cloud technologies.

One of the business challenges of Kubernetes in the cloud architecture is the ongoing cost ($300-$600/month per resource) during the learning or development process. To lower this ongoing cost per resource, we focused on a method to use on-prem Kubernetes deployments.


We have found examples online of using minikube and Oracle Virtualbox to assist with keeping costs low while using an on-prem deployment but did not find many examples of using Vmware Workstation to our satisfaction. Our goal was to utilize a solution that we are very familiar with and has the supporting capabilities for rollback via snapshots.

We have used Vmware Workstation for many years while working on service projects. We cannot overstate its usefulness to offer a “play-ground” and development environment independent of a client’s environment. The features of snapshots allow for negative use-case testing or “what-if” scenarios to destroy or impact solutions being tested with minimal impact.

In this entry, we will discuss the use of Vmware Workstation and CentOS (or Ubuntu) as the primary Kubernetes Nodes. Both CentOS and/or Ubuntu OS are used by the cloud providers as their Kubernetes nodes, so this on-prem process will translate well.

Some of our team members run the Kubernetes environment from their laptop, a collection of individual servers, or a larger server that may scale to the number of vCPU/RAM required for the Kubernetes solution.

Decision 1: Choose an OS to be used.

Either CentOS or Ubuntu OS is acceptable to use for on-prem. When we checked the OSes used by the cloud providers, we noted they used one of these two (2) OS for Linux OS. We decided on CentOS 7, as iptables for routing are used within Kubernetes; and iptables are used by default in CentOS 7. You may find that other OSes will work fine as well.

Decision 2: Build a reference image

Identify all expected binaries to be used within this image. This reference image will be cloned for the Kubernetes control plane node (1) and the worker nodes (3-4). We will also use this image to build a supporting node (non-Kubernetes) for SiteMinder integration and a docker repository for the Kubernetes docker images. For a total of six (6) nodes.

Decision 3: DNS and Certificates

Recommendation: Please do not attempt to deploy a Kubernetes solution on-prem without having purchased a DNS domain/site and use wild card certificates tied to the DNS domain.

Without these two (2) supporting components, it is a challenge to have a working Kubernetes solution that reflects what you will experience in a cloud deployment.

For example, we purchased a domain for $12/year, and then created several “A” records that will host the IP addresses we may use to redirect to cloud or on-prem. Using sub-domains “A” records, we can have as many cloud addresses as we wish.

DNS "A" Records Example:    
aks.iam.anapartner.net (MS Azure),  
eks.iam.anapartner.net (Amazon),  
gke.iam.anapartner.net (Google).      

DNS "CNAME" Records Example:  
alertmanager.aks.iam.anapartner.net, 
grafana.aks.iam.anapartner.net, 
jaeger.aks.iam.anapartner.net,
kibana.aks.iam.anapartner.net, 
mgmt-ssp.aks.iam.anapartner.net, 
sm.aks.iam.anapartner.net, 
ssp.aks.iam.anapartner.net.       
Example of using Synology DNS Server for Kubernetes cluster’s application. With “A” and “CNAME” records.

Finally, we prefer to use wildcard certificates for these domains to avoid challenges within our Kubernetes deployment. There are several services out there offering free certificates.

We chose Let’sEncrypt https://letsencrypt.org/. While Let’sEncrypt has automated processes to renew their certs, we chose to use their DNS validation process with a CertBot solution. We can renew these certificates every 90 days for on-prem usage. The DNS validation process requires a unique string generated by the Let’sEncrypt process to be populated in a DNS “TXT” record like so: _acme-challenge.aks.iam.anapartner.net . See the example at the bottom of this blog entry on this process.

Decision 4: Supporting Components: Storage, Load-Balancing, DNS Resolution (Local)

The last step required for on-prem deployment is where will you decide to place persistence storage for your Kubernetes cluster. We chose to use an NFS share.

We first tested using the control-plane node, then decided to move the NFS share to a Synology NAS solution. Similar for the DNS resolution option, at first we used a DNS service on the control-plane node and then moved to to the Synology NAS solution.

For Load-Balancing, Kubernetes has a service option of NodePort and LoadBalancing. The LoadBalancing service if not deployed in the cloud, will default to NodePort behavior. To introduce load balancing for on-prem, we introduced the HA-proxy service on the control-plane node, along with Kubernetes NodePort service to meet this goal.

After the decisions have been made, we can now walk through the steps to set up a Vmware environment for Kubernetes.

Reference Image

Step 1: Download the OS DVD ISO image for deployment on Vmware Workstation (Centos 7 / Ubuntu ).

Determine specs for the future solution to be deployed on Kubernetes. Some solutions have pods that may require minimal memory/disc space. For the solution we decided on deploying, we confirmed that we need 16 GB RAM and 4vCPU minimal. We have confirmed these specs were required by previously deploying the solution in a cloud environment.

Without these memory/cpu specs, the solution that we chose would pause the deployment of Kubernetes pods to the nodes. You may or may not see error messages in the deployment of pods stating that the nodes did not have enough resources for all or some of the pods.

For disc size, we selected 100 GB to future-proof the solution during testing. For networking, please select BRIDGED mode, to allow the Vmware images to have minimal network issues when routing within your local network. Please avoid double NAT’ing the deployment to reduce your headaches.

Step 2: Install useful base packages and disable any UI tools. Please install an Entropy Daemon to avoid delays due to certificates usage of /dev/random and low entropy.

### UI Update for CentOS7 was stopping yum deployment - not required for our solution to be tested (e.g. VIP Auth Hub)
# su to root to run the below commands.   We will add sudo access later.

su - 
systemctl disable packagekit; systemctl stop packagekit; systemctl status packagekit

### Installed base useful packages.

yum -y install dnf epel-release yum-utils nfs-utils 

### Install useful 2nd tools.

yum -y install openldap-clients jq python3-pip tree

pip3 install yq
yum -y upgrade


### Install Entropy process (epel repo)

dnf -y install haveged
systemctl enable haveged --now

Step 3: Install docker and update the docker configuration for use with Kubernetes. Update the path & storage-driver for the docker images for initial deployment.

Ref: https://docs.docker.com/storage/storagedriver/overlayfs-driver/

### Install Docker repo & docker package

yum-config-manager --add-repo  https://download.docker.com/linux/centos/docker-ce.repo
dnf -y install docker-ce
docker version
systemctl enable docker --now
docker version

### Update docker image info after deployment and restart service

cat << EOF > /etc/docker/daemon.json
{
"debug": false,
"data-root": "/home/docker-images",
"exec-opts": ["native.cgroupdriver=systemd"],
"log-driver": "json-file",
"log-opts": {
"max-size": "100m"
},
"storage-driver": "overlay2"
}
EOF

### Restart docker to load updated image info.
systemctl restart docker; systemctl status docker; docker version

Step 4: Deploy the three (3) primary Kubernetes & the HELM binaries.

Ensure you select a Kubernetes version that matches what solution you wish to deploy and work with. This can be a gotcha if the Kubernetes binaries update during a dnf / yum upgrade process and your solution has not been vetted for the newer release of Kubernetes. See the reference link below on how to upgrade Kubernetes binaries.

Ref: https://kubernetes.io/docs/tasks/administer-cluster/kubeadm/kubeadm-upgrade/

### Add k8s repo

cat << EOF > /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=https://packages.cloud.google.com/yum/repos/kubernetes-el7-x86_64
enabled=1
gpgcheck=1
repo_gpgcheck=1
gpgkey=https://packages.cloud.google.com/yum/doc/yum-key.gpg https://packages.cloud.google.com/yum/doc/rpm-package-key.gpg
exclude=kubelet kubeadm kubectl
EOF

### When upgrading the OS, be sure to use the correct version of kubernetes (remove and add) - Example to force version 1.20.11 ###

dnf upgrade -y
dnf remove -y kubelet kubeadm kubectl
dnf install -y kubelet-1.20.11-0.x86_64 kubeadm-1.20.11-0.x86_64 kubectl-1.20.11-0.x86_64 --disableexcludes=kubernetes


### Start the k8s process.

systemctl enable kubelet --now;  systemctl status kubelet
systemctl daemon-reload && systemctl enable kubelet --now
yum-config-manager --save --setopt=kubernetes.skip_if_unavailable=true

### Add HELM binary 

curl -fsSL -o get_helm.sh https://raw.githubusercontent.com/helm/helm/master/scripts/get-helm-3
chmod 700 get_helm.sh
./get_helm.sh

Step 5: OS configurations required or useful for Kubernetes. Kubernetes kubelet binary requires SWAP to be disabled.

Ref: https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/install-kubeadm/

### Stop FirewallD - May add ports later for security

systemctl stop firewalld;systemctl disable firewalld; iptables -F

### Update OS Parameters for kubernetes

setenforce 0
sed -i --follow-symlinks 's/SELINUX=enforcing/SELINUX=disabled/g' /etc/sysconfig/selinux
modprobe br_netfilter

cat << EOF > /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
EOF
sysctl --system

### Note:  IP forwarding is enabled by default.

sysctl -a | grep -i forward

### Note: Update /etc/fstab to comment out swap line with # character
### Warning:  kubectl init will fail if swap is left on cp or any worker node.

swapoff -a
sed -i 's|UUID\=\(.*\)-\(.*\)-\(.*\)-\(.*\)-\(.*\) swap|#UUID\=\1-\2-\3-\4-\5 swap|g' /etc/fstab
cat /etc/fstab

Step 6: Create SSH key for root or other services IDs to allow remote script updates from CP to Worker Nodes

### Create SSH key for root to allow remote script updates from CP to Worker Nodes - Enter a Blank/Null PASSWORD.

su - 
rm -rf ~/.ssh; echo y | ssh-keygen -b 4096  -C $USER -f ~/.ssh/id_rsa

### Copy the public rsa key to authorized keys to avoid password between cp/worker nodes for remote ssh commands.

cp -r -p ~/.ssh/id_rsa.pub ~/.ssh/authorized_keys;chmod 600 ~/.ssh/authorized_keys;ls -lart .ssh

### Test for remote connection with no password:   
  
ssh -i ~/.ssh/id_rsa  root@localhost    

### Copy the id_rsa key to your host system for ease of testing.

### Add your local non-root user to sudo wheel group.  Change vip to your user ID.

LOCALUSER=vip
gpasswd -a $LOCALUSER wheel

### Update sudoers file to allow wheel group with no-password

sed -i 's|# %wheel|%wheel|g' /etc/sudoers

###  View update wheel group.

grep "%wheel" /etc/sudoers

# Example of return query.
# %wheel  ALL=(ALL)       ALL
# %wheel  ALL=(ALL)       NOPASSWD: ALL

Step 7: Stop or adjust the OS network manager, shutdown the reference image, and create a Vmware Snapshot

### Adjust or Disable the OS NetworkManager (to avoid overwriting /etc/resolv.conf)
### Important when using an internal DNS server.

systemctl disable NetworkManager;systemctl stop NetworkManager

### reboot CentOS7 Image and validate no issues upon reboot.
reboot

### Shutdown image and manually create snapshot called  "base"

Vmware Workstation Cloning

Step 8: Now that we have a reference image, we can now make clone images for the control-plane (1), the worker nodes (4), and the supporting node (1). This is a fairly quick process.

export BASE=/home/me/vmware/kub
export REF=/home/me/vmware/kub/CentOS7/CentOS7.vmx

VM=cp;mkdir       -p $BASE/$VM; time vmrun -T ws clone $REF $BASE/$VM/$VM.vmx -cloneName=$VM -snapshot=base full
VM=worker01;mkdir -p $BASE/$VM; time vmrun -T ws clone $REF $BASE/$VM/$VM.vmx -cloneName=$VM -snapshot=base full
VM=worker02;mkdir -p $BASE/$VM; time vmrun -T ws clone $REF $BASE/$VM/$VM.vmx -cloneName=$VM -snapshot=base full
VM=worker03;mkdir -p $BASE/$VM; time vmrun -T ws clone $REF $BASE/$VM/$VM.vmx -cloneName=$VM -snapshot=base full
VM=worker04;mkdir -p $BASE/$VM; time vmrun -T ws clone $REF $BASE/$VM/$VM.vmx -cloneName=$VM -snapshot=base full
VM=sm;mkdir -p $BASE/$VM; time vmrun -T ws clone $REF $BASE/$VM/$VM.vmx -cloneName=$VM -snapshot=base full

Step 9: Start the clone images and remotely assign new hostname/IP addresses to the images

# Start cloned images for CP and Worker Nodes - Update any files as needed. 
 
export DOMAIN=aks.iam.anapartner.net
export PASSWORD_VM=Password01

### Start the cloned images for CP and Worker Nodes.

VM=cp;vmrun -T ws start $BASE/$VM/$VM.vmx nogui
VM=worker01;vmrun -T ws start $BASE/$VM/$VM.vmx nogui
VM=worker02;vmrun -T ws start $BASE/$VM/$VM.vmx nogui
VM=worker03;vmrun -T ws start $BASE/$VM/$VM.vmx nogui
VM=worker04;vmrun -T ws start $BASE/$VM/$VM.vmx nogui
VM=sm;vmrun -T ws start $BASE/$VM/$VM.vmx nogui
vmrun -T ws list | sort -rn


### Update Hostnames for CP and Worker Nodes with Domain.

VM=cp;vmrun -T ws -gu root -gp $PASSWORD_VM runScriptInGuest $BASE/$VM/$VM.vmx  /bin/bash "hostnamectl set-hostname $VM.$DOMAIN" -noWait
VM=worker01;vmrun -T ws -gu root -gp $PASSWORD_VM runScriptInGuest $BASE/$VM/$VM.vmx  /bin/bash "hostnamectl set-hostname $VM.$DOMAIN" -noWait
VM=worker02;vmrun -T ws -gu root -gp $PASSWORD_VM runScriptInGuest $BASE/$VM/$VM.vmx  /bin/bash "hostnamectl set-hostname $VM.$DOMAIN" -noWait
VM=worker03;vmrun -T ws -gu root -gp $PASSWORD_VM runScriptInGuest $BASE/$VM/$VM.vmx  /bin/bash "hostnamectl set-hostname $VM.$DOMAIN" -noWait
VM=worker04;vmrun -T ws -gu root -gp $PASSWORD_VM runScriptInGuest $BASE/$VM/$VM.vmx  /bin/bash "hostnamectl set-hostname $VM.$DOMAIN" -noWait
VM=sm;vmrun -T ws -gu root -gp $PASSWORD_VM runScriptInGuest $BASE/$VM/$VM.vmx  /bin/bash "hostnamectl set-hostname $VM.$DOMAIN" -noWait


### Update IP Address and Domain for NIC (ifcfg-ens33)

export CP=192.168.2.60
export WK1=192.168.2.61
export WK2=192.168.2.62
export WK3=192.168.2.63
export WK4=192.168.2.64
export SM=192.168.2.65

VM=cp;vmrun -T ws -gu root -gp $PASSWORD_VM runScriptInGuest $BASE/$VM/$VM.vmx  /bin/bash  "sed -i 's|TYPE=\"Ethernet\"|TYPE=\"Ethernet\"\nIPADDR=$CP\nDOMAIN=$DOMAIN|g'   /etc/sysconfig/network-scripts/ifcfg-ens33" -noWait
VM=worker01;vmrun -T ws -gu root -gp $PASSWORD_VM runScriptInGuest $BASE/$VM/$VM.vmx  /bin/bash  "sed -i 's|TYPE=\"Ethernet\"|TYPE=\"Ethernet\"\nIPADDR=$WK1\nDOMAIN=$DOMAIN|g'   /etc/sysconfig/network-scripts/ifcfg-ens33" -noWait
VM=worker02;vmrun -T ws -gu root -gp $PASSWORD_VM runScriptInGuest $BASE/$VM/$VM.vmx  /bin/bash  "sed -i 's|TYPE=\"Ethernet\"|TYPE=\"Ethernet\"\nIPADDR=$WK2\nDOMAIN=$DOMAIN|g'   /etc/sysconfig/network-scripts/ifcfg-ens33" -noWait
VM=worker03;vmrun -T ws -gu root -gp $PASSWORD_VM runScriptInGuest $BASE/$VM/$VM.vmx  /bin/bash  "sed -i 's|TYPE=\"Ethernet\"|TYPE=\"Ethernet\"\nIPADDR=$WK3\nDOMAIN=$DOMAIN|g'   /etc/sysconfig/network-scripts/ifcfg-ens33" -noWait
VM=worker04;vmrun -T ws -gu root -gp $PASSWORD_VM runScriptInGuest $BASE/$VM/$VM.vmx  /bin/bash  "sed -i 's|TYPE=\"Ethernet\"|TYPE=\"Ethernet\"\nIPADDR=$WK4\nDOMAIN=$DOMAIN|g'   /etc/sysconfig/network-scripts/ifcfg-ens33" -noWait
VM=sm;vmrun -T ws -gu root -gp $PASSWORD_VM runScriptInGuest $BASE/$VM/$VM.vmx  /bin/bash  "sed -i 's|TYPE=\"Ethernet\"|TYPE=\"Ethernet\"\nIPADDR=$SM\nDOMAIN=$DOMAIN|g'   /etc/sysconfig/network-scripts/ifcfg-ens33" -noWait

Step 10: Enable the network gateway, disable DHCP, and reboot the images

export DOMAIN=aks.iam.anapartner.net
export PASSWORD_VM=Password01

### Update to create a new default GATEWAY HOST to address routing issues to external IP addresses.
GATEWAY=192.168.2.1

VM=cp;vmrun -T ws -gu root -gp $PASSWORD_VM runScriptInGuest $BASE/$VM/$VM.vmx  /bin/bash  "sed -i 's|# Created by anaconda|# Created by anaconda\nGATEWAY=$GATEWAY|g' /etc/sysconfig/network" -noWait
VM=worker01;vmrun -T ws -gu root -gp $PASSWORD_VM runScriptInGuest $BASE/$VM/$VM.vmx  /bin/bash  "sed -i 's|# Created by anaconda|# Created by anaconda\nGATEWAY=$GATEWAY|g' /etc/sysconfig/network" -noWait
VM=worker02;vmrun -T ws -gu root -gp $PASSWORD_VM runScriptInGuest $BASE/$VM/$VM.vmx  /bin/bash  "sed -i 's|# Created by anaconda|# Created by anaconda\nGATEWAY=$GATEWAY|g' /etc/sysconfig/network" -noWait
VM=worker03;vmrun -T ws -gu root -gp $PASSWORD_VM runScriptInGuest $BASE/$VM/$VM.vmx  /bin/bash  "sed -i 's|# Created by anaconda|# Created by anaconda\nGATEWAY=$GATEWAY|g' /etc/sysconfig/network" -noWait
VM=worker04;vmrun -T ws -gu root -gp $PASSWORD_VM runScriptInGuest $BASE/$VM/$VM.vmx  /bin/bash  "sed -i 's|# Created by anaconda|# Created by anaconda\nGATEWAY=$GATEWAY|g' /etc/sysconfig/network" -noWait
VM=sm;vmrun -T ws -gu root -gp $PASSWORD_VM runScriptInGuest $BASE/$VM/$VM.vmx  /bin/bash  "sed -i 's|# Created by anaconda|# Created by anaconda\nGATEWAY=$GATEWAY|g' /etc/sysconfig/network" -noWait

### Disable DHCP (to avoid overwriting /etc/resolv.conf)

VM=cp;vmrun -T ws -gu root -gp $PASSWORD_VM runScriptInGuest $BASE/$VM/$VM.vmx  /bin/bash  "sed -i 's|BOOTPROTO=\"dhcp\"|BOOTPROTO=\"none\"|g'   /etc/sysconfig/network-scripts/ifcfg-ens33" -noWait
VM=worker01;vmrun -T ws -gu root -gp $PASSWORD_VM runScriptInGuest $BASE/$VM/$VM.vmx  /bin/bash  "sed -i 's|BOOTPROTO=\"dhcp\"|BOOTPROTO=\"none\"|g'   /etc/sysconfig/network-scripts/ifcfg-ens33" -noWait
VM=worker02;vmrun -T ws -gu root -gp $PASSWORD_VM runScriptInGuest $BASE/$VM/$VM.vmx  /bin/bash  "sed -i 's|BOOTPROTO=\"dhcp\"|BOOTPROTO=\"none\"|g'   /etc/sysconfig/network-scripts/ifcfg-ens33" -noWait
VM=worker03;vmrun -T ws -gu root -gp $PASSWORD_VM runScriptInGuest $BASE/$VM/$VM.vmx  /bin/bash  "sed -i 's|BOOTPROTO=\"dhcp\"|BOOTPROTO=\"none\"|g'   /etc/sysconfig/network-scripts/ifcfg-ens33" -noWait
VM=worker04;vmrun -T ws -gu root -gp $PASSWORD_VM runScriptInGuest $BASE/$VM/$VM.vmx  /bin/bash  "sed -i 's|BOOTPROTO=\"dhcp\"|BOOTPROTO=\"none\"|g'   /etc/sysconfig/network-scripts/ifcfg-ens33" -noWait
VM=sm;vmrun -T ws -gu root -gp $PASSWORD_VM runScriptInGuest $BASE/$VM/$VM.vmx  /bin/bash  "sed -i 's|BOOTPROTO=\"dhcp\"|BOOTPROTO=\"none\"|g'   /etc/sysconfig/network-scripts/ifcfg-ens33" -noWait

 
### Reboot VIP Auth Hub CP and Nodes 

VM=cp;vmrun -T ws -gu root -gp $PASSWORD_VM runScriptInGuest $BASE/$VM/$VM.vmx  /bin/bash  "reboot" -noWait
VM=worker01;vmrun -T ws -gu root -gp $PASSWORD_VM runScriptInGuest $BASE/$VM/$VM.vmx  /bin/bash  "reboot" -noWait
VM=worker02;vmrun -T ws -gu root -gp $PASSWORD_VM runScriptInGuest $BASE/$VM/$VM.vmx  /bin/bash  "reboot" -noWait
VM=worker03;vmrun -T ws -gu root -gp $PASSWORD_VM runScriptInGuest $BASE/$VM/$VM.vmx  /bin/bash  "reboot" -noWait
VM=worker04;vmrun -T ws -gu root -gp $PASSWORD_VM runScriptInGuest $BASE/$VM/$VM.vmx  /bin/bash  "reboot" -noWait
VM=sm;vmrun -T ws -gu root -gp $PASSWORD_VM runScriptInGuest $BASE/$VM/$VM.vmx  /bin/bash  "reboot" -noWait

Step 11: Update DNS on the clone images remotely using vmrun

### Update /etc/resolv.conf for correct DNS server.
### Ensure DHCP and Network Manager are disable to prevent these services from overwrite behavior.

export DOMAIN=aks.iam.anapartner.net
export PASSWORD_VM=Password01
DNSNEW=192.168.2.20

VM=cp;vmrun -T ws -gu root -gp $PASSWORD_VM runScriptInGuest $BASE/$VM/$VM.vmx  /bin/bash  "echo 'nameserver $DNSNEW' >>  /etc/resolv.conf" -noWait
VM=worker01;vmrun -T ws -gu root -gp $PASSWORD_VM runScriptInGuest $BASE/$VM/$VM.vmx  /bin/bash  "echo 'nameserver $DNSNEW' >>  /etc/resolv.conf" -noWait
VM=worker02;vmrun -T ws -gu root -gp $PASSWORD_VM runScriptInGuest $BASE/$VM/$VM.vmx  /bin/bash  "echo 'nameserver $DNSNEW' >>  /etc/resolv.conf" -noWait
VM=worker03;vmrun -T ws -gu root -gp $PASSWORD_VM runScriptInGuest $BASE/$VM/$VM.vmx  /bin/bash  "echo 'nameserver $DNSNEW' >>  /etc/resolv.conf" -noWait
VM=worker04;vmrun -T ws -gu root -gp $PASSWORD_VM runScriptInGuest $BASE/$VM/$VM.vmx  /bin/bash  "echo 'nameserver $DNSNEW' >>  /etc/resolv.conf" -noWait
VM=sm;vmrun -T ws -gu root -gp $PASSWORD_VM runScriptInGuest $BASE/$VM/$VM.vmx  /bin/bash  "echo 'nameserver $DNSNEW' >>  /etc/resolv.conf" -noWait
 
 
### Reboot VIP Auth Hub CP and Nodes
 
VM=cp;vmrun -T ws -gu root -gp $PASSWORD_VM runScriptInGuest $BASE/$VM/$VM.vmx  /bin/bash  "reboot" -noWait
VM=worker01;vmrun -T ws -gu root -gp $PASSWORD_VM runScriptInGuest $BASE/$VM/$VM.vmx  /bin/bash  "reboot" -noWait
VM=worker02;vmrun -T ws -gu root -gp $PASSWORD_VM runScriptInGuest $BASE/$VM/$VM.vmx  /bin/bash  "reboot" -noWait
VM=worker03;vmrun -T ws -gu root -gp $PASSWORD_VM runScriptInGuest $BASE/$VM/$VM.vmx  /bin/bash  "reboot" -noWait
VM=worker04;vmrun -T ws -gu root -gp $PASSWORD_VM runScriptInGuest $BASE/$VM/$VM.vmx  /bin/bash  "reboot" -noWait
VM=sm;vmrun -T ws -gu root -gp $PASSWORD_VM runScriptInGuest $BASE/$VM/$VM.vmx  /bin/bash  "reboot" -noWait

Step 12: Copy the root .ssh public cert to your main host, rename it to a useful name and these test your newly deployed clone images for DSN resolution using ssh. Please confirm this step is successful prior to continuing with the configuration of the control plane and worker nodes.

### Copy the root id_rsa file to host system to allow ease of testing with ssh.

export CP=192.168.2.60
export WK1=192.168.2.61
export WK2=192.168.2.62
export WK3=192.168.2.63
export WK4=192.168.2.64
export SM=192.168.2.65

### Add the hosts for ssh pre-validation. 

ssh-keyscan -p 22 $CP >> ~/.ssh/known_hosts
ssh-keyscan -p 22 $WK1 >> ~/.ssh/known_hosts
ssh-keyscan -p 22 $WK2 >> ~/.ssh/known_hosts
ssh-keyscan -p 22 $WK3 >> ~/.ssh/known_hosts
ssh-keyscan -p 22 $WK4 >> ~/.ssh/known_hosts
ssh-keyscan -p 22 $SM >> ~/.ssh/known_hosts


### Rename from id_rsa to vip_kub_root_id_rsa

ssh -tt -i ~/vip_kub_root_id_rsa root@$CP 'cat /etc/resolv.conf'
ssh -tt -i ~/vip_kub_root_id_rsa root@$WK1 'cat /etc/resolv.conf'
ssh -tt -i ~/vip_kub_root_id_rsa root@$WK2 'cat /etc/resolv.conf'
ssh -tt -i ~/vip_kub_root_id_rsa root@$WK3 'cat /etc/resolv.conf'
ssh -tt -i ~/vip_kub_root_id_rsa root@$WK4 'cat /etc/resolv.conf'
ssh -tt -i ~/vip_kub_root_id_rsa root@$SM 'cat /etc/resolv.conf'


### Validate Access with ssh to CP and Worker Nodes new IP addresses.

FQDN=ssp.aks.iam.anapartner.net
ssh -tt -i ~/vip_kub_root_id_rsa root@$CP  "ping -c 2 $FQDN"
ssh -tt -i ~/vip_kub_root_id_rsa root@$WK1 "ping -c 2 $FQDN"
ssh -tt -i ~/vip_kub_root_id_rsa root@$WK2 "ping -c 2 $FQDN"
ssh -tt -i ~/vip_kub_root_id_rsa root@$WK3 "ping -c 2 $FQDN"
ssh -tt -i ~/vip_kub_root_id_rsa root@$WK4 "ping -c 2 $FQDN"
ssh -tt -i ~/vip_kub_root_id_rsa root@$SM "ping -c 2 $FQDN"

Update CP (controlplane) Node

Step 13a: Copy files to CP Node from Vmware Workstation host and configure the CP node for dedicated CP usage. Recommend using two terminals/sessions to speed up the process. Install HAproxy for Load Balancing, copy the Let’s Encrypt wild card certificates, and copy the Kubernetes solution you will be deploying (scripts/yaml).

### Open Terminal 1 to CP host.
### Add bash completion to have better use of TAB to view parameters.

CP=192.168.2.60
ssh -tt -i ~/vip_kub_root_id_rsa root@$CP
dnf -y install bash-completion
echo 'export KUBECONFIG=/etc/kubernetes/admin.conf'  >>~/.bashrc
kubectl completion bash >/etc/bash_completion.d/kubectl
echo "alias k=kubectl | complete -F __start_kubectl k" >>~/.bashrc

### Install HAProxy and replace the haproxy.cfg file.
dnf -y install haproxy
systemctl enable haproxy --now
netstat -anp | grep -i -e haproxy

### Open Terminal 2 to host and push files to CP node.
### Copy HAProxy configuration, certs, and scripts
scp -i ~/vip_kub_root_id_rsa  haproxy.cfg   root@$CP:/etc/haproxy/haproxy.cfg
scp -i ~/vip_kub_root_id_rsa  cloud-certs-aks-eks-gke_exp-202X-01-12.tar  root@$CP:
scp -i ~/vip_kub_root_id_rsa  202X-11-03_vip_auth_hub_working_centos7_v2.tar   root@$CP:

### On Terminal 1 - on CP host - Restart to use new haproxy configuration file.
systemctl restart haproxy
netstat -anp | grep -i -e haproxy

### Extract CERTS to root home folder
tar -xvf cloud-certs-aks-eks-gke_exp-202X-01-12.tar

### Extract Working Scripts 
tar -xvf 202X-11-03_vip_auth_hub_working_centos7_v2.tar

### Update env variables for unique environment within step00 file.
vi step00_kubernetes_env.sh

### Add the env variables to the .bashrc file
echo ". ./step00_kubernetes_env.sh"

Step 13b: Example of /etc/haproxy/haproxy.cfg configuration for Kubernetes Load Balancing functionality for on-prem worker nodes. HAproxy deployed on control plane (CP) node. The example configuration file will route TCP 80/443/389 to one (1) of the four (4) worker nodes. If a Kubernetes NodePort service is enabled for TCP 389 (31888) ports, then this load balancer will function correctly and route the traffic for LDAP traffic as well.

[root@cp ~]# cat /etc/haproxy/haproxy.cfg
global
    user haproxy
    group haproxy
    chroot /var/lib/haproxy
    log /dev/log    local0
    log /dev/log    local1 notice
defaults
    mode http
    log global
    retries 2
    timeout http-request 10s
    timeout queue 1m
    timeout connect 10s
    timeout client 10m
    timeout server 10m
    timeout http-keep-alive 10s
    timeout check 10s
    maxconn 3000
frontend ingress
    bind *:80
    option tcplog
    mode http
    option forwardfor
    option http-server-close
    default_backend kubernetes-ingress-nodes
backend kubernetes-ingress-nodes
    mode http
    balance roundrobin
    server k8s-ingress-0 worker01.aks.iam.anapartner.net:80 check fall 3 rise 2 send-proxy-v2
    server k8s-ingress-1 worker02.aks.iam.anapartner.net:80 check fall 3 rise 2 send-proxy-v2
    server k8s-ingress-2 worker03.aks.iam.anapartner.net:80 check fall 3 rise 2 send-proxy-v2
    server k8s-ingress-2 worker04.aks.iam.anapartner.net:80 check fall 3 rise 2 send-proxy-v2
frontend ingress-https
    bind *:443
    option tcplog
    mode tcp
    option forwardfor
    option http-server-close
    default_backend kubernetes-ingress-nodes-https
backend kubernetes-ingress-nodes-https
    mode tcp
    balance roundrobin
    server k8s-ingress-0 worker01.aks.iam.anapartner.net:443 check fall 3 rise 2 send-proxy-v2
    server k8s-ingress-1 worker02.aks.iam.anapartner.net:443 check fall 3 rise 2 send-proxy-v2
    server k8s-ingress-2 worker03.aks.iam.anapartner.net:443 check fall 3 rise 2 send-proxy-v2
    server k8s-ingress-2 worker04.aks.iam.anapartner.net:443 check fall 3 rise 2 send-proxy-v2
frontend ldap
    bind *:389
    option tcplog
    mode tcp
    default_backend kubernetes-nodes-ldap
backend kubernetes-nodes-ldap
    mode tcp
    balance roundrobin
    server k8s-ldap-0 worker01.aks.iam.anapartner.net:31888  check fall 3 rise 2
    server k8s-ldap-1 worker02.aks.iam.anapartner.net:31888  check fall 3 rise 2
    server k8s-ldap-2 worker03.aks.iam.anapartner.net:31888  check fall 3 rise 2
    server k8s-ldap-2 worker04.aks.iam.anapartner.net:31888  check fall 3 rise 2

Deploy Solution on Kubernetes

Step 14: Validate that DNS and Storage are ready before deploying any solution or if you wish to have a base Kubernetes environment to use with the control-plane and four (4). worker nodes.

### Step:  Setup NFS Share either on-prem remote server or Synology NFS
### Use version 4.x checkbox for Synology.

### Example of lines on remote Linux Host with NFS share.

yum -y install nfs-utils
systemctl enable --now nfs-server rpcbind
mkdir -p /export/nfsshare ; chown nobody /export/nfsshare ; chmod -R 777 /export/nfsshare
echo "/export/nfsshare *(rw,sync,no_root_squash,insecure)" >> /etc/exports
exportfs -rav; exportfs -v

firewall-cmd --add-service=nfs --permanent
firewall-cmd --add-service={nfs3,mountd,rpc-bind} --permanent 
firewall-cmd --reload 



#### Setup DNS entries (A and CNAME) for twelve (12) items ( May be on-prem DNS or Synology DNS)

ns.aks.iam.anapartner.net  A  IP_ADDRESS (192.168.2.60)
aks.iam.anapartner.net  NS ns.aks.iam.anapartner.net
cp.aks.iam.anapartner.net  A  IP_ADDRESS (192.168.2.60)
worker01.aks.iam.anapartner.net  A  IP_ADDRESS (192.168.2.61)
worker02.aks.iam.anapartner.net  A  IP_ADDRESS (192.168.2.62)
worker03.aks.iam.anapartner.net  A  IP_ADDRESS (192.168.2.63)
worker04.aks.iam.anapartner.net  A  IP_ADDRESS (192.168.2.64)
sm.aks.iam.anapartner.net  A  IP_ADDRESS (192.168.2.65)
kibana CNAME cp.aks.iam.anapartner.net 
grafana CNAME cp.aks.iam.anapartner.net 
jaeger CNAME cp.aks.iam.anapartner.net 
alertmanager CNAME cp.aks.iam.anapartner.net 
ssp CNAME cp.aks.iam.anapartner.net 
ssp-mgmt CNAME cp.aks.iam.anapartner.net 

### Pre-Step:  Enable DNS resolution for external IP addresses
### Enable forwarding to external h/w router and 8.8.8.8

Step 15: Recommendation. Deploy your solution in steps using Kubernetes yaml or Helm charts to assist with debugging any deployment issues. Do not forget to use kubectl logs, and kubectl describe to isolate startup or cert issues.

### Run scripts one-by-one.  They will have a watch command in each that will 
### provide feedback on the startup processes.
### Total startup from scratch to final with VIP Sample App is about 15-20 minutes.
### Note:  Step04 has a different chart variables for on-prem for Symantec Directory.
### Note:  /step00_kubernetes_env.sh is called by each script.


./step01_kubernetes_cluster_init_with_worker_nodes.sh
./step02_kubernetes_cluster_with_ingress_and_other_charts.sh
./step03_kubernetes_cluster_with_vip_auth_hub_charts.sh
./step04_kubernetes_cluster_with_vip_auth_hub_sample_app.sh

Docker Registry for On-Prem

There are two (2) types of docker registries we have found useful.

a. The standard Mirror method will capture all docker images from “docker.io” site to a local mirror. When Kubernetes or Helm deployments are used, the docker configuration file can be adjusted to check the local mirror without updating Kubernetes yaml files or Helm charts.

b. The second method is a full query of all images after they have been deployed once, and using the docker push process into a local registry. The challenge of the second method is that the Kubernetes yaml files and/or Helm charts do have to be updated to use this local registry.

Either method will help lower bandwidth cost to re-download the same docker images, if you use a docker prune method to keep your worker nodes disc size “clean”. If the docker prune process is not used, you may notice that the worker nodes may run out of disc space due to temporary docker images/containers that did not clean up properly.

#!/bin/bash
#################################################################################
#  Create a local docker mirror registry for docker-ios
#  and local docker non-mirror registry for all other images
#  to minimize download impact
#  during restart of the kubernetes solution
#
#  All registry iamges will be placed on NFS share
#  mount -v -t nfs 192.168.2.30:/volume1/nfs /mnt  &>/dev/null
#
# Certs will be provided by Let's Encrypt every 90 days
#
#  For docker-io mirror registry, all clients must have the following line in
#  /etc/docker/daemon.json     {Note:  Use commas as needed}
#
#    "registry-mirrors":
#     [
#      "https://sm.aks.iam.anapartner.net:444"
#     ],
#
#
#
# ANA 11/2021
#
#################################################################################
# To remove all containers - to allow restart of process
docker rm -f `docker ps -a | grep -v -e CONTAINER | awk '{print $1}'` ; docker image rm `docker image ls | grep -v -e REPOSITORY | grep -e minutes -e hour -e days -e '2 weeks'|  awk '{print $3}'` &>/dev/null


#################################################################################
# Update HOST name for local server for docker image
HOST=sm.aks.iam.anapartner.net
NFS_SERVER=192.168.2.30
NFS_SHARE=/volume1/nfs


#################################################################################
function start_registry {

    local_port=$1
    remote_registry_name=$2

    if [ "$3" == "" ]; then
        remote_registry_url=$remote_registry_name
    else
        remote_registry_url=$3
    fi

    echo -e "$local_port $remote_registry_name $remote_registry_url"


mount -v -t nfs $NFS_SERVER:$NFS_SHARE /mnt  &>/dev/null
mkdir -p /mnt/registry/${remote_registry_name}  &>/dev/null

docker run -d --name registry-${remote_registry_name}-mirror  \
-p $local_port:443 \
--restart=always \
-e REGISTRY_HTTP_ADDR=0.0.0.0:443 \
-e REGISTRY_PROXY_REMOTEURL="https://${remote_registry_url}/" \
-e REGISTRY_HTTP_TLS_CERTIFICATE=/certs/fullchain.pem \
-e REGISTRY_HTTP_TLS_KEY=/certs/privkey.pem \
-e REGISTRY_COMPATIBILITY_SCHEMA1_ENABLED=true \
-v /mnt/registry/certs:/certs \
-v /mnt/registry/${remote_registry_name}:/var/lib/registry \
registry:latest

sleep 1
echo "#################################################################################"
curl -s -X GET  https://$HOST:$local_port/v2/_catalog | jq
echo "#################################################################################"

}

#################################################################################
# start_registry <local_port>    <remote_registry_name>  <remote_registry_url>
#################################################################################

start_registry   444             docker-io               registry-1.docker.io

#################################################################################
# Non-Proxy configuration to allow 'docker tag & docker push' for all other images
#################################################################################

remote_registry_name=all
local_port=455
mkdir -p /var/lib/docker/registry/${remote_registry_name}  &>/dev/null
docker run -d --name registry-${remote_registry_name}-mirror  \
-p $local_port:443 \
--restart=always \
-e REGISTRY_HTTP_ADDR=0.0.0.0:443 \
-e REGISTRY_HTTP_TLS_CERTIFICATE=/certs/fullchain.pem \
-e REGISTRY_HTTP_TLS_KEY=/certs/privkey.pem \
-e REGISTRY_COMPATIBILITY_SCHEMA1_ENABLED=true \
-v /mnt/registry/certs:/certs \
-v /mnt/registry/${remote_registry_name}:/var/lib/registry \
registry:latest

sleep 1
echo "#################################################################################"
curl -s -X GET  https://$HOST:$local_port/v2/_catalog | jq
echo "#################################################################################"
docker ps -a
echo "#################################################################################"

echo "##### To tail the log of the docker-io container - useful for monitoring helm deployments  #####"
echo "docker logs `docker ps -a  --no-trunc | grep -v NAMES | grep 'docker-io' | awk '{print $1}'` -f "
echo "#################################################################################"
echo "##### To tail the log of the ALL container - useful for monitoring helm deployments  #####"
echo "docker logs `docker ps -a  --no-trunc | grep -v NAMES | grep 'all' | awk '{print $1}'` -f  "
echo "#################################################################################"
echo "##### Location of Registry Files on NFS share #####"
echo "ls -lart /mnt/registry/docker-io/docker/registry/v2/repositories"
echo "ls -lart /mnt/registry/all/docker/registry/v2/repositories"
echo "#################################################################################"

Example of the /etc/docker/daemon.json configuration file to use a local mirror for docker.io. See the parameter of “registry-mirrors”. Unfortunately, we were unable to use this process for the other docker registries.

{
"debug": false,
"data-root": "/home/docker-images",
"exec-opts": ["native.cgroupdriver=systemd"],
"storage-driver": "overlay2",
"registry-mirrors":
[
"https://sm.aks.iam.anapartner.net:444"
],
"log-driver": "json-file",
"log-opts": {
"max-size": "100m"
}
}

Let’s Encrypt Certbot and DNS validation

Use Let’sEncrypt Certbox and manual DNS validation, to create our 90-day wild card certificates. Manual DNS validation allows us to avoid setting up a public-facing component for our internal labs.

Ref: https://letsencrypt.org/docs/challenge-types/

# Step 1:  Install SNAP service for Certbot usage on your host OS

cat /etc/redhat-release
Red Hat Enterprise Linux release 8.3 (Ootpa)

sudo yum install -y  snapd
Updating Subscription Management repositories.
Package snapd-2.49-2.el8.x86_64 is already installed.

systemctl enable --now snapd.socket

### Wait 1 min

snap install core; sudo snap refresh core



# Step 2: Remove prior certbot (if installed by yum/dnf)

yum remove -y certbot.


# Step 3:  Install new "classic" Certbot

sudo snap install --classic certbot
certbot 1.17.0 from Certbot Project (certbot-eff✓) installed

sudo ln -s /snap/bin/certbot /usr/bin/certbot



# Step 4: Issue certbot command with wildcard cert & update your DNS TXT record with the string provided.


sudo certbot certonly --manual  --preferred-challenges dns -d *.aks.iam.anapartner.org --register-unsafely-without-email

Saving debug log to /var/log/letsencrypt/letsencrypt.log

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Please read the Terms of Service at
https://letsencrypt.org/documents/LE-SA-v1.2-November-15-2017.pdf. You must
agree in order to register with the ACME server. Do you agree?
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
(Y)es/(N)o: Y
Account registered.
Requesting a certificate for *.aks.iam.anapartner.org

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Please deploy a DNS TXT record under the name:

_acme-challenge.iam.anapartner.org.

with the following value:

u2cXXXXXXXXXXXXXXXXXXXXc

Before continuing, verify the TXT record has been deployed. Depending on the DNS
provider, this may take some time, from a few seconds to multiple minutes. You can
check if it has finished deploying with aid of online tools, such as the Google
Admin Toolbox: https://toolbox.googleapps.com/apps/dig/#TXT/_acme-challenge.iam.anapartner.org.
Look for one or more bolded line(s) below the line ';ANSWER'. It should show the
value(s) you've just added.

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

# Step 5:  In a 2nd terminal, validate that the DNS record has been updated and can be seen by a standard DNS query.   Have the 2nd console window open to test the DNS record, prior to <ENTER> key on verification request

# Example:
nslookup -type=txt _acme-challenge.aks.iam.anapartner.org
Non-authoritative answer:
_acme-challenge.aks.iam.anapartner.org  text = "u2cXXXXXXXXXXXXXXXXXXXXc"


# Step 6:  Press <ENTER> after you have validated the TXT record.

Press Enter to Continue
Waiting for verification...
Cleaning up challenges
Subscribe to the EFF mailing list (email: nala@baugher.us).

IMPORTANT NOTES:
 - Congratulations! Your certificate and chain have been saved at:
   /etc/letsencrypt/live/aks.iam.anapartner.org/fullchain.pem
   Your key file has been saved at:
   /etc/letsencrypt/live/aks.iam.anapartner.org/privkey.pem
  


# Step 7: View certs of fullchain.pem & privkey.pem  

cat /etc/letsencrypt/live/aks.iam.anapartner.org/fullchain.pem
-----BEGIN CERTIFICATE-----

<REMOVED>
-----END CERTIFICATE-----
-----BEGIN CERTIFICATE-----
<REMOVED>
-----END CERTIFICATE-----

cat /etc/letsencrypt/live/aks.iam.anapartner.org/privkey.pem
-----BEGIN PRIVATE KEY-----

<REMOVED>
-----END PRIVATE KEY-----




# Step 8:  Use the two files for your kubernetes solution 

# Step 9:  Ensure domain on host OS, cp, worker nodes in /etc/resolv.conf is set correctly to aks.iam.anapartner.org    to allow the certs to be resolved correctly.

# Step 10:  Ensure Synology NAS DNS service is configurated with all alias 


# Step 11:  Optional: Validate certs with openssl


# Show the kubernetes self-signed cert

true | openssl s_client -connect kibana.aks.iam.anapartner.org:443 2>/dev/null | openssl x509 -inform pem -noout -text

# Show the new wildcard cert for same hostname &  port

curl -vvI  https://kibana.aks.iam.anapartner.org/app/home#/

curl -vvI  https://kibana.aks.iam.anapartner.org/app/home#/   2>&1 | awk 'BEGIN { cert=0 } /^\* SSL connection/ { cert=1 } /^\*/ { if (cert) print }'

nmap -p 443 --script ssl-cert kibana.aks.iam.anapartner.org


Kubernetes Side Note:   Let's Encrypt certs do NOT show up within the Kubernetes cluster certs check process.

kubeadm certs check-expiration

View of the DNS TXT records to be updated with your DNS service provider. The Let’sEncrypt Certbot will need to be able to query these records for it to assign you wildcard certificates. Create the _acme-challenge hostname entry as a TXT type, and paste in the string provided by the Let’sEncrypt Certbot process. Wait 5 minutes or test the TXT record with nslookup, then upon positive validation, continue the Let’sEncrypt Certbot process.

View your kubernetes cluster / nodes for any constraints

After your cluster is created and you have worker nodes joined to the cluster, you may wish to monitor for any constraints of your on-prem deployment. Kubectl command with the action verb of describe or top is very useful for this goal.

kubectl describe nodes worker01
kubectl top node / kubectl top pod

Kubernetes Training (Formal)

If you are new to Kubernetes, we recommend the following class. You may need to dedicate 4-8 weeks to complete the course and then take the CKA exam via the Linux Foundation.

https://www.udemy.com/course/certified-kubernetes-administrator-with-practice-tests/ .

Kubernetes.io site has most of the information you need to get started.

https://kubernetes.io/docs/reference/kubectl/cheatsheet/

vApp IMPS PAM AD Authentication “Jump Server”

The virtual appliance and standalone deployment of Symantec (CA) Identity Suite allow for redirecting authentication for the J2EE tier application through Symantec SSO or directly to an Active Directory domain, instead of the existing userstore for the solution.

Challenge:

The standalone deployment of Symantec (CA) Identity Suite on MS Windows OS allowed for the mid-tier component to utilize PAM modules to redirect to AD authentication for the Global User.

However this PAM feature does not exist for Provisioning Servers on the virtual appliance.

To be clear, there are no expectations this feature will be introduced in the future roadmap for the solution, as the primary UI will be the web browser.

Review:

Symantec (CA) Identity Suite architecture for virtual appliance versus standalone deployment architecture.

  • The standalone deployment architecture has both MS Windows and Linux components of all tiers.
  • The vApp deployment architecture has primary Linux components and few MS Windows components.
    • The vApp MS Windows components do not include the IMPS (Provisioning Server)

Proposal:

To address this requirement of enabling AD authentication to the vApp Provisioning Server, we will introduce the concept of a “jump server”.

The “jump server” will utilize the standalone deployment of Symantec Identity Provisioning Server on an MS Windows OS. This “jump server” will be deployed as an “alternative server” integrated into the existing vApp Provisioning Directory deployment.

We will select deployment configuration ONLY of the Provisioning Server itself. We do not require the embedded CCS Service.

We will integrate this “jump server” deployment with the existing Symantec Identity solution.

Ensure the imps_datakey encryption seed file is in sync between all components vApp and standalone.

To avoid impacting the existing vApp deployment, we will NOT integrate the “jump server” deployment to the IME. The IME’s Directory XML for the Provisioning Directory will not be updated.

Important Note: The Symantec/CA Directory solution is required as a pre-step.

Summary of deployment steps:

  • Select a MS Window OS workstation (clean or with JCS/CCS Services) that may be part of the MS AD Domain
    • Option 1: [RECOMMENDED & PREFERRED] If using a clean OS, install MS .NetFramework 3.5.1 for the provisioning component.
      • Open cmd as administrator to deploy:  DISM /Online /Enable-Feature /All /FeatureName:NetFx3
    • Option 2: [MED-HIGH RISK] If using “side-deployment” on an existing JCS/CCS server (MS Win OS), we will need to make modifications to this server.
      • Will need to rename the file  C:\Windows\vpd.properties   to avoid conflict with the JCS/CCS component naming convention in this “registry” file. (see below screen shot)
      • Will require a post-install execution of the IMPS pwdmgr tool to address an MS Registry path conflict between the CCS and IMPS components.
  • Ensure all CA Directory hostnames are in DNS or in the MS Windows local host file (C:\Windows\System32\drivers\etc\hosts ) otherwise this “jump server” deployment will fail when it tries to validate all possible directory nodes’ hostnames and build the respective Directory knowledge files.
  • Create a reference file for the new IMPS router dxc file on at least one of the existing vApp Identity Suite Directory Server otherwise this “jump server” deployment will fail due to trust issue when testing connections to other directory nodes’ hostnames.
  • Deploy Symantec/CA Directory (if not already done) – default configurations. Otherwise, you will see this error message
  • Deploy IMPS MS Windows – Only IMPS (no CCS) with Alternative Server Selection Configuration & update to latest CP patches. Note: For “side-deployment” only: If the vpd.properties file was not renamed, then a name collision will occur due to this registry file, if using the JCS/CCS server to side-deploy. It is low risk to change this file, as it is used to prevent deployments of lower release version of components over the prior installed higher release versions of the same component. If there is a concern, all components can be reinstalled as needed. Do not forget to install the latest CP patches to ensure this “jump server” is the same binary level as the vApp solution.
  • Review of additional notes during deployment of “jump server”. Note: For “side-deployment” only: On the page that ask for the Identity Suite Directory connection information, you will see the solution attempt to load env variables that do not exist. Override these value and enter the Directory hostname, Port 20394, and the default bind DN credentials for a Directory userID: eTDSAContainerName=DSAs,eTNamespaceName=CommonObjects,dc=im,dc=etadb
  • Deploy IMPM Manager GUI if needed.
  • Post-Deployment – Update IMPM Manager GUI preference to ONLY connect to the new IMPS server on MS Windows. Use the “Enable Failover” checkbox and place the IP/hostname first in the list. Hint: Remove the other IMPS servers from this list or you may add an extra digit to the IMPS servers entries you wish to save, but prevent auto-connectivity to them. Confirm able to authenticate directly through the solution using prior credentials for your service ID: etaadmin or imadmin. This will validate connectivity to the existing vApp Identity Suite solution.
  • On the “jump server” under the Provisioning Server\pam\ADS folder copy the etapam.dll to the IMPS \bin folder. Then copy the etapam_id.conf configuration file to the \pam parent folder. Update the parameters in this file. Set the enable= parameter to yes. Set the domain= to either the MS AD Domain or use the FQDN hostname of the ADS Domain Controller (DC). If we use the FQDN hostname of the DC the “jump server” does NOT have to be made a member of the MS AD Domain. Save the file and restart the “CA Identity Manager – Provisioning Server”
  • Validate PAM functionality in the IMPS etatrans log is enabled. We will see two (2) entries: PAM: Initialization started (same for all use-cases) and PAM: Not enabled or No PAM managed endpoint. We want “PAM: No PAM managed endpoint” – That is an extra feature we could enable, but do not require for the “jump server” scenario.
  • Validate PAM functionality with MS Sysinternals. Ensure that we copied the etapam.dll to the bin folder and that the configuration file is being read.
  • Test authentication using IMPM Manager login as IMPS Manager Global User that has same userID format as AD sAMAccountName. Negative Use-Case testing: Create a new AD domain user that does NOT exist as a Global User and attempt to authentication. Test with etaadmin or other Global User that does NOT have a match AD sAMAcccount name entry. Review the IMPS etatrans logs on the “jump server”

Update the IMPS encrypted seed file imps_datakey as needed.

Note: The MS Win version of IMPS encrypted seed file may be different than the vApp seed.

If this step is skipped, there will be no obvious error message with the exception that a bind has failed for communication to the JCS/CCS services.

After this file is updated, we will need to re-install IMPS service to ensure that all prior encryption passwords are replaced with new passwords using the new seed file. Basically, we need to install the MS Win version of IMPS Server twice, e.g. standard install, change the seed file value, re-install with update all components and updated passwords.

CCS Service conflict with “side-loading” IMPS Service {“side-loading” methodology}

The “side loading “process of deploying the “jump server” IMPS Provisioning Server on the JCS/CCS Server will impact starting of the CCS service. The installation will update the MS Registry with extra branches and updated shared attribute values between the CCS service and IMPS service, e.g. ETAHOME.

This challenge is a strong reason why we may choose the “clean” installation methodology, to avoid this conflict and possible support challenge.

To address this concern, update the new registry values that store the embedded reversible encrypted password for the CCS Service. Use the password reset tool “pwdmgr” and reset the “Connector Server” for both “eta” & “im” domain to the prior stored password. If the imps_datakey file is not in sync between all provisioning servers (& ccs service), then we will see failed bind connections error messages in the logs.

We will now be able to stop/start the JCS service, and see the embedded CCS service stop and start as well.

Example of challenge and error messages if imps_datakey is not updated and in sync.

Use the following command, csfconfig.exe, under the newly deployed IMPS bin folder to view the JCS connectors defined to the solution stack.

C:\Program Files (x86)\CA\Identity Manager\Provisioning Server\bin>csfconfig.exe auth=etaadmin show
EtaSSL.initialize: CRYPTO_library_init: 1
EtaSSL.initialize: SSL_library_init: 1
Enter your authentication password:


C:\Program Files (x86)\CA\Identity Manager\Provisioning Server\bin>echo Password01 > c:\imps.pwd


C:\Program Files (x86)\CA\Identity Manager\Provisioning Server\bin>csfconfig.exe auth=etaadmin add name=pamjcs host=192.168.242.143 pass=c:\imps.pwd br-add=@ debug=yes port=20411
EtaSSL.initialize: CRYPTO_library_init: 1
EtaSSL.initialize: SSL_library_init: 1
Enter your authentication password:
Created CS object with name = pamjcs

C:\Program Files (x86)\CA\Identity Manager\Provisioning Server\bin>csfconfig.exe auth=etaadmin remove name=pamjcs 
EtaSSL.initialize: CRYPTO_library_init: 1
EtaSSL.initialize: SSL_library_init: 1
Enter your authentication password:

We will see both error status when the imps_datakey file is out-of-sync with others. Please ensure the Linux & MS Win versions are in sync.

You may view the file imps_datakey being referenced with the pwdmgr tool:

su - imps
strace -e trace=open,openat  pwdmrg

open(“/opt/CA/IdentityManager/ProvisioningServer/data/tls/keymgmt/imps_datakey”, O_RDONLY) = 5

You wish to monitor what accounts (embedded) are updated with the IMPS pwdmgr tool: su – imps and execute the two commands in a different SSH shell to monitor the pwdmgr.log that was enabled.

eta-env action=set name=eta_pwdmgr_log type=string value=true
tail -f $ETAHOME/bin/pwdmgr.log

Enablement of extra functionality (bypass the no-sync option on Global User password update)

You may wish to keep the Global User and AD password in sync. If they are not, then you will have two passwords that will work for the Global User account. The newer PAM AD authentication credentials, and the older Global User password. The etapam.dll module data path appears to check for PAM AD first, and if it fails, then it will check the Global User eTPassword field as well.

Enable the AD endpoint in the etapam_id.conf file. The type and domain will be as shown, e.g. Active Directory and im (for the vApp). The endpoint-name will be free-form and whatever you may have named your AD endpoint in the IMPS GUI.

Monitor the startup of the PAM module within the IMPS etatrans*.log

Perform a use-case test with changing a Global User account without correlation to an AD endpoint; and then retest with a Global User that is correlated to an AD endpoint. Do both test with NO SYNC operation

If the Global User is already correlated to an AD endpoint account, then we will see a “Child Modify” operation to the correlated AD endpoint account’s Password within the IMPS etatrans*.log.

One “gotcha”. There appears to be a check against the AD password policy. If the new password does not fit the AD password policy, the following error message will appear, “ETA_E_0007 <MGU>, Global user XXXXXXX modification failed: PAM account password updated failed: Account password must match global user password.

The hidden cost of Entropy to your business

On Linux OS, there are two (2) device drivers that provide entropy “noise” for components that require encryption, e.g. the /dev/random and the /dev/urandom device drivers. The /dev/random is a “blocking” device driver. When the “noise” is low, any component that relies on this driver will be “stalled” until enough entropy is returned. We can measure the entropy from a range of 0-4096. Where a value over 1000 is excellent. Any value in the double or single digits will impact the performance of the OS and solutions with delays. The root cause of these delays is not evident during troubleshooting, and typically there are no warning nor error messages related to entropy.

watch -n 1 cat /proc/sys/kernel/random/entropy_avail

The Symantec Identity Suite solution, when deployed on Linux OS is typically deployed with the JVM switch -Djava.security.egd=file:/dev/./urandom for any component that uses Java (Oracle or AdoptOpenJDK), e.g. Wildfly (IM/IG/IP) and IAMCS (JCS). This JVM variable is sufficient for most use-cases to manage the encryption/hash needs of the solution.

However, for any component that does not provide a mechanism to use the alternative of /dev/urandom driver, the Linux OS vendors offer tools such as the “rng-tools” package. We can review what OS RNGD service is available using package tools, e.g.

dnf list installed | grep -i rng

If the Symantec Identity Suite or other solutions are deployed as standalone components, then we may adjust the Linux OS as we need with no restrictions to add the RNGD daemon as we wish. One favorite is the HAVEGED daemon over the default OS RNGD.

See prior notes on value and testing for Entropy on Linux OS (standalone deployments):

https://community.broadcom.com/enterprisesoftware/communities/community-home/digestviewer/viewthread?GroupId=2197&MID=720771&CommunityKey=f9d65308-ca9b-48b7-915c-7e9cb8fc3295&tab=digestviewer

https://community.broadcom.com/HigherLogic/System/DownloadDocumentFile.ashx?DocumentFileKey=7747b411-2e1e-4bc2-8284-9b8856790ef9

Challenge for vApp

The challenge for Virtual Appliances is that we are limited to what functionality the Symantec Product Team provides for us to leverage. The RNGD service was available on the vApp r14.3, but was disabled for OS challenges with 100% utilization with CentOS 6.4. The service is still installed, but the actual binary is non-executable.

https://knowledge.broadcom.com/external/article/97774/ca-identity-suite-low-entropy-on-virtual.html
https://knowledge.broadcom.com/external/article/139759/ca-identity-suite-142-vapp-rngd-proces.html
https://broadcom-stage.adobecqms.net/us/en/symantec-security-software/identity-security/identity-suite/14-3/virtual-appliance/administering-virtual-appliance/using-the-login-shell.html

A new Virtual Appliance patch would be required to re-enable this RNGD on vApp r14.3cp2. We have access via sudo, to /sbin/chkconfig, /sbin/service to re-enable this service, but as the binary is not executable, we cannot progress any further. We can see the alias in the documentation still exist, but the OS alias was removed in the cp2 update.

However, since vApp r14.4 was release, we can focus on this Virtual Appliance which is running Centos 8 stream. The RNGD service here is disabled (masked) but can be re-enabled for our use with the sudo command. There is no current documented method for RNGD on vApp r14.4 at this time, but the steps below will show an approved way using the ‘config’ userID and sudo commands.

Confirm that the “rng-tools” package is installed and that the RNGD binary is executable. We can also see that the RNGD service is “masked”. Masked services are prevented from starting manually or automatically as an extra safety measure when we wish for tighter control over our systems.

If we test OS entropy for this vApp r14.4 server without RNGD, we can monitor how a simple BASH shell script that emulates a password being generated will impact the “entropy” of /dev/random. The below script will reduce the entropy to low numbers. This process will now impact the OS itself and any components that reference /dev/random. We can observe with “lsof /dev/random” that the java programs will still reference /dev/random; even though most activity is going to /dev/urandom.

Using the time command in the BASH shell script, we can see that the response is rapid for the first 20+ iterations, but as soon as the entropy is depleted, each execution is delayed by 10-30x times.

counter=1;MAX=100;while [ $counter -le $MAX ]; do echo "##########  $counter ##########" ; time dd if=/dev/random bs=8 count=1 2> /dev/null | base64; counter=$(( $counter + 1 )); done;

Enable RNGD on vApp r14.4 & Testing

Now let’s see what RNGD service will do for us when it is enabled. Let’s follow the steps below to unmask, enable, and start the RNGD service as the ‘config’ userID. We have access to sudo to the Centos 8 Stream command of /sbin/systemctl.

sudo /usr/bin/systemctl status rngd.service
ls -lart /etc/systemd/system/rngd.service
sudo /usr/bin/systemctl unmask rngd.service
sudo /usr/bin/systemctl enable rngd.service
cat /usr/lib/systemd/system/rngd.service
sudo /usr/bin/systemctl start rngd.service
sudo /usr/bin/systemctl status rngd.service
ps -ef | grep rngd | grep -v grep

After the RNGD service is enabled, test again with the same prior BASH shell script but bump the loops to 1000 or higher. Note using the time command we can see that each loop finishes within a fraction of a second.

counter=1;MAX=1000;while [ $counter -le $MAX ]; do echo "##########  $counter ##########" ; time dd if=/dev/random bs=8 count=1 2> /dev/null | base64; counter=$(( $counter + 1 )); done;

Summary

Aim to keep the solution footprint small and the right-sized to solve the business’ needs. Do not accept the default performance; avoid over-purchasing to scale to your expected growth.

Use the JVM switch wherever there is a java process, e.g. BLC or home-grown ETL (extract-transform-load) processes.

-Djava.security.egd=file:/dev/./urandom

If you suspect a dependence may impact the OS or other processes on /dev/random, then enable the OS RNGD and perform your testing. Monitor with the top command to ensure RNGD service is providing value and not impacting the solution.