While preparing to enable a feature within the Identity Suite Virtual Appliance for TLS encryption for the Provisioning Tier to send notification events, we noticed some challenges that we wish to clarify.
The Identity Suite Virtual Appliance has four (4) web services that use pre-built self-signed certificates when first deployed. Documentation is provided to change these certificates/key using aliases or soft-links.
One of the challenges we discovered is the Provisioning Tier may be using an older version of libcurl & OpenSSL that have constraints that need to be managed. These libraries are used during the web submission to the IME ETACALLBACK webservice. We will review the processes to capture these error messages and how to address them.
We will introduce the use of Let’s Encrypt wildcard certificates into the four (4) web services and the Provisioning Server’s ETACALLBACK use of a valid public root certificate.
The Apache HTTPD service is used for both a forward proxy (TCP 443) to the three (3) Wildfly Services and service for the vApp Management Console (TCP 10443). The Apache HTTPD service SSL certs use the path /etc/pki/tls/certs/localhost.crt for a self-signed certificate. A soft-link is used to redirect this to a location that the ‘config’ service ID has access to modify. The same is true for the private key.
/etc/pki/tls/certs/localhost.crt -> /opt/CA/VirtualAppliance/custom/apache-ssl-certificates/localhost.crt
/etc/pki/tls/private/localhost.key -> /opt/CA/VirtualAppliance/custom/apache-ssl-certificates/localhost.keyA view of the Apache HTTPD SSL self-signed certificate and key.
The three (3) Wildfly services are deployed for the Identity Manager, Identity Governance and Identity Portal components. The configuration for TLS security is defined within the primary Wildfly configuration file of standalone.xml. The current configuration is already setup with the paths to PKCS12 keystore files of:
/opt/CA/VirtualAppliance/custom/wildfly-ssl-certificates/caim-srv
/opt/CA/VirtualAppliance/custom/wildfly-ssl-certificates/caig-srv
/opt/CA/VirtualAppliance/custom/wildfly-ssl-certificates/caip-srvA view of the three (3) Wildfly PKCS12 keystore files and view of the self-signed cert/key with the pseudo hostname of the vApp host.
Provisioning Server process for TLS enablement for IME ETACALLBACK process.
Step 1. Ensure that the Provisioning Server is enabled to send data/notification events to the IME.
Step 2. Within the IME Management Console, there is a baseURL parameter. This string is sent down to the Provisioning Server upon restart of the IME, and appended to a list. This list is viewable and manageable within the Provisioning Manager UI under [System/Identity Manager Setup]. The URL string will be appended with the string ETACALLBACK/?env=identityEnv. Within this Provisioning Server, we can manage which URLs have priority in the list. This list is a failover list and not load-balancing. We have the opportunity to introduce an F5 or similar load balancer URL, but we should enable TLS security prior.
Step 3. Added the public root CA Cert or CA chain certs to the following location. [System/Domain Configuration/Identity Manager Server/Trusted CA Bundle]. This PEM file may be placed in the Provisioning Server bin folder with no path or may use a fully qualified path to the PEM file. Note: The Provisioning Server is using a version of openssl/libcurl that will report errors that can be managed with wildcard certificates. We will show the common errors in this blog entry.
Let’sEncrypt https://letsencrypt.org/ Certificates
Let’sEncrypt Certificates offers a free service to build wildcard certificates. We are fond of using their DNS method to request a wildcard certificate.
sudo certbot certonly --manual --preferred-challenges dns -d *.aks.iam.anapartner.dev --register-unsafely-without-emailLet’s Encrypt will provide four (4) files to be used. [certN.pem, privkeyN.pem, chainN.pem, fullchainN.pem]
cert1.pem [The primary server side wildcard cert]
privkey1.pem [The primary server side private key associated with the wildcard cert]
chain1.pem [The intermediate chain certs that are needed to validate the cert1 cert]
fullchain1.pem [two files together in the correct order of cert1.pem and chain1.pem.]
NOTE: fullchain1.pem is the file you typically would use as the cert for a solution, so the solution will also have the intermediate CA chain certs for validation]
Important Note: One of the root public certs was cross-signed by another root public cert that expired. Most solutions are able to manage this challenge, but the provisioning service ETACALLBACK has a challenge with an expired certificate, but there are replacements for this expired certificate that we will walk through. Ref: https://letsencrypt.org/docs/dst-root-ca-x3-expiration-september-2021/
Create a new CA chain PEM files for LE (Let’s Encrypt) validation to use with the Provisioning Server.
CERT=lets-encrypt-r3.pem;curl -s -O -L https://letsencrypt.org/certs/$CERT ; openssl x509 -text -noout -in $CERT | grep -i -e issue -e not -e subject ; ls -lart $CERT
CERT=isrgrootx1.pem;curl -s -O -L https://letsencrypt.org/certs/$CERT ; openssl x509 -text -noout -in $CERT | grep -i -e issue -e not -e subject ; ls -lart $CERT
CERT=isrg-root-x2.pem;curl -s -O -L https://letsencrypt.org/certs/$CERT ; openssl x509 -text -noout -in $CERT | grep -i -e issue -e not -e subject ; ls -lart $CERT
cat lets-encrypt-r3.pem isrgrootx1.pem isrg-root-x2.pem > combine-chain-letsencrypt.pem
Replacing the certificates for the vApp Apache, Wildfly (3), and Provisioning Server (ETACALLBACK)
Apache HTTPD Service (TCP 443/10443) (May need to reboot vApp)
cp -r -p /home/config/aks.iam.anapartner.dev/fullchain2.pem /opt/CA/VirtualAppliance/custom/apache-ssl-certificates/localhost.crt
cp -r -p /home/config/aks.iam.anapartner.dev/privkey2.pem /opt/CA/VirtualAppliance/custom/apache-ssl-certificates/localhost.keyWildfly Services (TCP 8443/8444/84445) for IM, IG, and IP (restart services after update)
openssl pkcs12 -export -inkey /home/config/aks.iam.anapartner.dev/privkey2.pem -in /home/config/aks.iam.anapartner.dev/fullchain2.pem -out /opt/CA/VirtualAppliance/custom/wildfly-ssl-certificates/caim-srv -password pass:changeit
restart_im
openssl pkcs12 -export -inkey /home/config/aks.iam.anapartner.dev/privkey2.pem -in /home/config/aks.iam.anapartner.dev/fullchain2.pem -out /opt/CA/VirtualAppliance/custom/wildfly-ssl-certificates/caig-srv -password pass:changeit
restart_ig
openssl pkcs12 -export -inkey /home/config/aks.iam.anapartner.dev/privkey2.pem -in /home/config/aks.iam.anapartner.dev/fullchain2.pem -out /opt/CA/VirtualAppliance/custom/wildfly-ssl-certificates/caip-srv -password pass:changeit
restart_ipProvisioning Server ETACALLBACK public certificate location (restart imps service) [Place in bin folder]
su - imps
cp -r -p /home/config/aks.iam.anapartner.dev/combine-chain-letsencrypt.pem /opt/CA/IdentityManager/ProvisioningServer/bin/
imps stop; imps startValidation of updated services.
Use openssl s_client to validate certificates being used. Examples below for TCP 443 and 8443
true | openssl s_client -connect vapp143.aks.iam.anapartner.dev:443 -CAfile combine-chain-letsencrypt.pem | grep "Verify return code"
true | openssl s_client -connect vapp143.aks.iam.anapartner.dev:8443 -CAfile combine-chain-letsencrypt.pem | grep "Verify return code"To view all certs in the chain, use the below openssl s_client command with -showcerts switch:
true | openssl s_client -connect vapp143.aks.iam.anapartner.dev:443 -CAfile combine-chain-letsencrypt.pem -showcerts
true | openssl s_client -connect vapp143.aks.iam.anapartner.dev:8443 -CAfile combine-chain-letsencrypt.pem -showcertsValidate with browsers and view the HTTPS lock symbol to view the certificate
Test with an update to a Provisioning Global User’s attribute [Note: No need to sync to accounts]. Ensure that the Identity Manager Setup Log Level = DEBUG to monitor this submission with the Provisioning Server etanotifyXXXXXXX.log.
A view of the submission for updating the Global User’s Description via IMPS (IM Provisioning Server) etanotifyXXXXXXX.log. The configuration will be loaded for using the URLs defined. Then we can monitor for the submission of the update.
Finally, a view using the IME VST (View Submitted Tasks) for the ETACALLBACK process using the task Provisioning Modify User.
Common TLS errors seen with the Provisioning Server ETACALLBACK
Ensure that the configuration is enabled for debug log level, so we may view these errors to correct them. [rc=77] will occur if the PEM file does not exist or is not in the correct path. [rc=51] will occur if the URL defined does not match the exact server-side certificate (this is a good reason to use a wildcard certificate or adjust your URL FQDN to match the cert subject (CN=XXXX) value. [rc=60] will occur if the remote web service is using a self-signed certificate or if the certificate has any expiration dates within the certificate or chain or the public root CA cert.
Other Error messages (curl)
If you see an error message with Apache HTTPD (TCP 443) with curl about “curl: (60) Peer certificate cannot be authenticated with known CA certificates”, please ignore this, as the vApp does not have the “ca-bundle.crt” configuration enabled. See RedHat note: https://access.redhat.com/solutions/523823
References
https://knowledge.broadcom.com/external/article?articleId=54198
https://community.broadcom.com/HigherLogic/System/DownloadDocumentFile.ashx?DocumentFileKey=849ea21f-cc5a-4eac-9988-465a75165cf1
https://curl.se/libcurl/c/libcurl-env.html
https://knowledge.broadcom.com/external/article/204213/how-to-setup-inbound-notifications-to-us.html
https://knowledge.broadcom.com/external/article/213480/how-to-replace-the-vapp-wildfly-ssl-cert.html https://www.stephenwagner.com/2021/09/30/sophos-dst-root-ca-x3-expiration-problems-fix/