google – ANA Technology Partner

Our goal is to move away from using self-signed certificates, often supplied by various software vendors and Kubernetes deployments, in favor of a unified approach with a single wildcard certificate covering multiple subdomains. This strategy allows for streamlined updates of a single certificate, which can then be distributed across our servers and projects as needed.

We have identified LetsEncrypt as a viable source for these wildcard certificates. Although LetsEncrypt offers a Docker/Podman image example, we have discovered an alternative that integrates with Google Domains. This method automates the entire process, including validation of the DNS TXT record and generation of wildcard certificates. It’s important to note that Google Domains operates independently from Google Cloud DNS, and its API is limited to updating TXT records only.

In this blog post, we will concentrate on how to validate the certificates provided by LetsEncrypt using OpenSSL. We will also demonstrate how to replace self-signed certificates with these new ones on a virtual appliance.

View the pubkey of the LetsEncrypt cert.pem and privkey.pem file to confirm they match

Note: LetsEncrypt private keys are now key-type = ECDSA by default. You may still request RSA key-type.

openssl x509 -noout -pubkey -in cert.pem

openssl pkey -pubout -in privkey.pem

When key-type = rsa, we can use either openssl validation method (rsa with md5 or pkey with pubout)

openssl x509 -noout -modulus -in cert.pem | openssl md5

openssl rsa  -noout -modulus -in privkey.pem | openssl md5

Validate DNS & Dates of the cert, chain, fullchain pem files

openssl x509 -text -noout -in cert.pem | grep -e DNS -e 'Not Before:' -e 'Not After :' -e 'Subject: C'

while openssl x509 -noout -text | grep -e Certificate: -e Issuer: -e DNS -e 'Not Before:' -e 'Not After :' -e 'Subject: C' ; do :; done < chain.pem 2>/dev/null

while openssl x509 -noout -text | grep -e Certificate: -e Issuer: -e DNS -e 'Not Before:' -e 'Not After :' -e 'Subject: C' ; do :; done < fullchain.pem 2>/dev/null

Download the current trusted public root CA cert from LetsEncrypt

We need to download this root CA cert for solutions, appliances, and on-prem Kubernetes Clusters that do NOT have these root CA certs in their existing keystores.

curl -sOL https://letsencrypt.org/certs/isrgrootx1.pem

Validate the cert.pem file with the public root CA cert and the provided chain.pem file

Will return an “OK” response if valid. CA certs order is important, if reversed this process will fail. Validation still fails if we only have chain.pem or fullchain.pem (see image below) without the correct public root CA cert from LetsEncrypt. Note: This public root CA cert is typically provided in updated modern browsers. Note2: While the fullchain.pem does have a CA cert with the CN = ISRG Root X1, this does NOT appear to be the correct one based on the error reported, so we have downloaded the correct CA cert to be used with the same CN name of ISRG Root X1 (see following images below)

openssl verify -CAfile <(cat isrgrootx1.pem chain.pem) cert.pem

Combine cert.pem with the public root CA cert and chain.pem for a complete chain cert in the CORRECT order.

Important Note: cert.pem MUST be first in this list otherwise validation will fail. Please note that there are two (2) root CA certs with the same CN, that may cause some confusion when validating the chain.

cat cert.pem isrgrootx1.pem chain.pem > combined_chain_with_cert.pem

Validate certs with openssl server process and two (2) terminal ssh sessions/windows

1st terminal session – run an openssl server (via openssl s_server) on port 9443 (any open port). The -www switch will send a status message back to the client when it connects. This includes information about the ciphers used and various session parameters. The output is in HTML format so this option will normally be used with a web browser.

openssl s_server -key privkey.pem -cert combined_chain_with_cert.pem -accept 9443 -www

2nd terminal session – run openssl s_client and curl with the combined chain cert to validate. Replace the FQDN with your LetsEnscrypt domain in the wildcard cert. Example below FQDN is training.anapartner.net. You may also use a browser to access the openssl s_server web server with a FQDN.

true | openssl s_client -connect localhost:9443 -CAfile combined_chain_with_cert.pem

curl -v --cacert combined_chain_with_cert.pem --resolve training.anapartner.net:9443:127.0.0.1 https://training.anapartner.net:9443

CERTBOT deployment via PODMAN

Example of using the official Certbot image with podman. We recommend using multiple -d switches with *.subdomain1.domain.com to allow a single cert be used for many of your projects. Reference of this deployment

podman run --rm -it \
-v "$(pwd)/etc-output:/etc/letsencrypt:z" \
-v "$(pwd)/var-output:/var/lib/letsencrypt:z" \
 certbot/certbot certonly --manual -d wildcard.domain.com \
-d *.subdomain1.domain.com \
-d *.subdomain2.domain.com \
-d *.subdomain3.domain.com \
-d *.subdomain4.domain.com

A version of podman with the Google Domain API TXT integration. We use variables for reuse of this code for various testing domains. This docker image will temporarily create the Google Domain TXT records via a REST API, that are needed for Certbot DNS validation, then the process will remove the TXT records. There is no manual interaction required. We use this process with a bash shell script to run as needed or via scheduled events.

podman run --rm \
  -v ${LETSENCRYPT}/var/lib/letsencrypt:/var/lib/letsencrypt:z \
  -v ${LETSENCRYPT}/etc/letsencrypt:/etc/letsencrypt:z \
  -v ${LETSENCRYPT}/var/log/letsencrypt:/var/log/letsencrypt:z \
  --cap-drop=all \
  ghcr.io/aaomidi/certbot-dns-google-domains:latest \
  certbot certonly  \
  --authenticator 'dns-google-domains' \
  --dns-google-domains-credentials /var/lib/letsencrypt/dns_google_domains_credentials_${domain}.ini \
  --non-interactive --agree-tos -m dns@${domain} \
  --server 'https://acme-v02.api.letsencrypt.org/directory' \
  --non-interactive \
  --dns-google-domains-zone "${domain}" \
  -d "${CN_OF_TLS_DOMAIN}" \
  -d "*.$subdomain01.$domain" \
  -d "*.$subdomain02.$domain" \
  -d "*.$subdomain03.$domain" \
  -d "*.$subdomain04.$domain" \
  -d "*.$subdomain05.$domain" \
  -d "*.$subdomain06.$domain" \
  -d "*.$subdomain07.$domain" \
  -d "*.$subdomain08.$domain" \
  -d "*.$subdomain09.$domain" \
  -d "*.$subdomain10.$domain" \
  -d "*.$subdomain11.$domain" \
  -d "*.$subdomain12.$domain"

Replace Identity Suite vApp Apache Certificate with LetsEncrypt

We see tech notes and an online document but wanted to provide a cleaner step by step process to update the Symantec IGA Virtual Appliance certificates for the embedded Apache HTTPD service under the path /opt/CA/VirtualAppliance/custom/apache-ssl-certificates

# Collect the generated LetsEncrypt certs via certbot, save them, scp to the vApp host, and then extract them

tar -xvf letsencrypt-20231125.tar


# View the certs

ls -lart


# Validate LetEncrypt cert via pubkey match between private key and cert

openssl x509 -noout -pubkey -in cert.pem
openssl pkey -pubout -in privkey.pem


# Download the latest LetsEncrypt public root CA cert

curl -sOL https://letsencrypt.org/certs/isrgrootx1.pem


# Validate a full chain with root with cert (Note: order is important on cat process)

openssl verify -CAfile <(cat isrgrootx1.pem chain.pem) cert.pem


# Create a full chain with root cert and LetsEncrypt chain in the correct ORDER

cat cert.pem isrgrootx1.pem chain.pem > combined_chain_with_cert.pem


# Move prior Apache HTTPD cert files

mv localhost.key localhost.key.original
mv localhost.crt localhost.crt.original


# Link the new LetsEncrypt files to same names of localhost.XXX

ln -s privkey.pem localhost.key
ln -s combined_chain_with_cert.pem localhost.crt


# Restart apache (httpd)

sudo systemctl restart httpd


# Test with curl with the FQDN name in the CN or SANs of the cert, e.g. training.anapartner.net

curl -v --cacert combined_chain_with_cert.pem --resolve training.anapartner.net:443:127.0.0.1 https://training.anapartner.net:443


# Test with browser with the FQDN name

Example of integration and information reported by browser

View of the certificate as shown with a CN (subject) = training.anapartner.net

A view of the SANS wildcard certs that match the FQDN used in the browser URL bar of iga.k8s-training-student01.anapartner.net

Example of error messages from Apache HTTPD service’s log files if certs are not in correct order or validate correctly. One message is a warning only, the other message is a fatal error message about the validation between the cert and private key do not match. Use the pubkey check process to confirm the cert/key match.

[ssl:warn] [pid 2206:tid 140533536823616] AH01909: CA_IMAG_VAPP:443:0 server certificate does NOT include an ID which matches the server name
[ssl:emerg] [pid 652562:tid 140508002732352] AH02565: Certificate and private key CA_IMAG_VAPP:443:0 from /etc/pki/tls/certs/localhost.crt and /etc/pki/tls/private/localhost.key do not match

Most Mobile Authenticator Apps will allow you to backup the Authenticator registration to an account.

Alternatively, if you have a spare phone (with or without a SIM chip), you may wish to deploy your Authenticator Apps to a 2nd phone, IPad, or Android Tablet to grant yourself additional freedom from being forced to using a single device for authentication.

Important Note: If the website allows it, you can register your QR code multiple times to different Authenticator Apps on the SAME or DIFFERENT phone. If you already registered to a site, you may re-register the QR code on both devices to ensure they both have the same “seed” for your login ID.

You may use your Ipad/Android Tablet without needing your primary phone near you while authenticating to your secure applications/websites.

Below is an example of using the following Authenticator Apps that registered the same QR code, e.g. Last Pass Authenticator (Red Shield Icon), Google Authenticator (Grey G), Microsoft Authenticator (Blue Lock Icon), and Okta Verify Authenticator (Blue “O” CheckMark Icon).

We did a test to confirm that these Authenticator Apps are all time based with your unique registration QR Code. As you can see from the below screenshot, any time-based authenticator app will return the same code within that 60 seconds cycle before they rotate.

Please note that other authenticators do not base the return value ONLY on time but other variables. Example: SecureID Token (Cloud Icon), Symantec VIP Access (Yellow Circle with Checkmark Icon), Okta Mobile (Blue Icon), and IRS2Go – Authenticator & App (IRS Logo Icon).

As we see more accounts get compromised, we strongly recommend using one or more of any authenticator applications with your mobile phone. Please note, all of these authenticator applications are free to use by the vendors.

Every website you access with an account usually has a “two-factor” authentication security setting that you may enable. You can enroll your mobile phone with the provided QR (quick response) code.

Examples of QR Codes that you may scan with your cell phone camera. Modern cell phone will auto transcribe these pictures into text for a web site URL, text, or registration code. The below three QR codes are all text base messages that you may practice your cell phone on. The more characters, the smaller the blocks will be in the QR code.

Hopefully, this entry may have value to you for account recovery, or managing access for/with a partner, spouse, dependents, and parents accounts.

Additional benefit, if the primary phone is lost or damaged, you will still have access to your accounts without being forced to go through recovery methods on each account, e.g. disable Authentication App, prove your identity, access your account, re-apply Authenticator App.

Only negative to this process is that you must remember to register 2nd device at the same time as the primary phone, for any new websites or wish to update your account on an existing website/application.

Example for Facebook TFA (Two-Factor-Authentication) Configuration:

Select Security and Login / Two-Factor Authentication under Facebook Settings. You will need to re-authenticate with your password to ensure that you are the correct person to change these settings.

https://www.facebook.com/security/2fac/settings

Next, select the “Authenticator App” Manage button to add in an Authenticator App. Have both your primary phone and your 2nd device available within one of the Authenticator App open. Scan the QA code with both devices. Do NOT click the Continue Button, until you have scanned with both devices. This QA code is the “seed” for your authentication app. If you have any issues, you can re-scan a new code to retry.

After you click continue, most application/websites will ask you to input the code from your phone/device into the website, to prove that it was recorded correctly. If you look at both devices, you should see the same code being repeated on both every 60 seconds when they rotate.

LassPass Example:

If you are a fan of LastPass, the online password management tool, you can enable the three (3) popular Authenticator apps as well. The Google Authenticator App selection may also be used with Okta Verify Authenticator App.