Tag: tls

Secure Application Introspection

Posted on by Leave a comment

Locate “the good, the bad, and the ugly” data with a transparent proxy.

Have you been frustrated with various enterprise/cloud solutions’ APIs implementation or documentation where a single case-sensitive data field entry delays progress? Does the solution have undocumented features for older client tools? Do you wish to know what your mobile apps or laptop sends to the internet?

Utilizing a proxy can help with all the above, and if the process is quick and straightforward, so much the better.

Typically for a proxy, there may be quite a bit of effort and steps. You may need to modify a client host/mobile phone to redirect web traffic with OS environmental variables of HTTP_PROXY and HTTPS_PROXY or adjustment of the underlying OS network/iptables. Prior, we typically set up the open-source Jmeter proxy with the OS environment variables to capture secure traffic data. This process works well for most applications. Additionally, the Firefox browser allows manual modification using a proxy without dependence on the OS environment settings if we wish to capture the user experience and any data challenges.

The example below of modifying a Firefox browser to use a “manual proxy configuration” instead of system/auto configurations.

To ensure accurate capture of web traffic submissions, a more thorough method is needed as the above process may fail if client tools or mobile apps cannot detect OS environmental variables.

We have found a perfect combination within the open-source tool of MITMproxy with podman and the embedded VPN feature of WireGuard.

The process in six (6) steps:

  1. Deployment of the WireGuard VPN client on the client host (MS Win/Linux/Mobile)
  2. Deployment of MITMproxy using podman (or docker) with WireGuard mode/configuration
  3. Edit the wireguard.conf file to have the correct public IP address and import this file to the WireGuard VPN client and establish the VPN connection.
  4. Copy the mitmproxy-ca-cert.cer to the client component Java or OS keystore (if needed) as a trusted CA cert.
  5. Open the MITMproxy Web UI or monitor the command line dashboard
  6. Execute your test on the client host and view the results in the MITMproxy Web UI for both request and response.

MITMproxy UI with WireGuard mode enabled.

The WireGuard client configuration will be provided in three (3) places: the MITMproxy logs (podman logs mitmproxy), the text file wireguard.conf (if podman/docker volumes are enabled), and the MITMproxy UI. The QR code is enabled for mobile phone use, but since the public IP address provided is not correct in this view, you will need to manually edit this configuration on your mobile phone during those use-cases to have the correct endpoint IP address.

MITMproxy UI with standard proxy configuration mode.

Bash Script:

Script to deploy MITMproxy with podman on a linux OS with two (2) configurations: Wireguard mode for any client applications that do not honor HTTP_PROXY/HTTPS_PROXY and Standard proxy mode. This bash script allows a shared volume to use the SAME certs to avoid managing different certs upon restart of the container.

#!/bin/bash
######################################################################################
#
#  Deploy MITMproxy with two (2) configurations:
#
#     MITMProxy with WireGuard mode enabled (UDP 51820) and Web UI (TCP 8081)
#     MITMProxy with standard proxy enabled (TCP 9080) and Web UI (TCP 9081)
#
#  Notes:  Use podman exec to check path and env variables
#    - Binaries:  dnf -y install podman 
#    - Use shared folder to avoid having two (2) different configuration files for both copies
#    - Do not forget the :z for -v volumes to avoid permissions issues
#    - Do not forget quotes around env -e variables
#    - Use --rm as needed
#    - Use this switch as needed, but do not leave it on:   --log-level debug \
#
#   Basic:  podman run -it -v /tmp/mitmproxy/:/home/mitmproxy/.mitmproxy:z -p 8080:8080 mitmproxy/mitmproxy
#   Logs:   podman logs mitmproxy-wireguard
#   Shell:  podman exec -it -u root mitmproxy bash
#
#  Options Ref.  https://docs.mitmproxy.org/stable/concepts-options/
#   - added stream_large_bodies=10m to lower impact to mitmproxy due
#       to possible large json/xml payloads 
#
#  ANA 07/2023
#
######################################################################################

MITMPROXY_HOMEPATH=/tmp/mitmproxy
echo ""
echo "You may delete the shared folder of ${MITMPROXY_HOMEPATH}"
echo "to remove prior configuration of mitmproxy certs & wireguard.conf files"
echo ""
#sudo rm -rf ${MITMPROXY_HOMEPATH}

mkdir -p ${MITMPROXY_HOMEPATH}
chmod -R 777 ${MITMPROXY_HOMEPATH}
ls -hlrt ${MITMPROXY_HOMEPATH}

echo ""
echo " Starting mitmproxy-wireguard proxy "
podman rm mitmproxy-wireguard -f  &>/dev/null
podman run -d -it --name mitmproxy-wireguard \
   -p 51820:51820/udp -p 8081:8081 \
   -l mitmproxy \
   -v ${MITMPROXY_HOMEPATH}:/home/mitmproxy/.mitmproxy:z  \
    docker.io/mitmproxy/mitmproxy \
    mitmweb --mode wireguard --ssl-insecure  --web-host 0.0.0.0 --web-port 8081 --set stream_large_bodies=10m


echo ""
echo " Starting mitmproxy-default proxy "
podman rm mitmproxy-default -f  &>/dev/null
podman run -d -it --name mitmproxy-default \
    -p 9080:9080 -p 9081:9081 \
    -l mitmproxy  \
    -v ${MITMPROXY_HOMEPATH}:/home/mitmproxy/.mitmproxy:z  \
     docker.io/mitmproxy/mitmproxy \
     mitmweb --set listen_port=9080 --web-host 0.0.0.0 --web-port 9081

echo ""
echo ""
echo "###############################################################################"
echo ""
echo " Running Podman Containers for MITMproxy"
sleep 5
podman ps -a --no-trunc | grep -i mitmproxy
echo ""
echo "###############################################################################"
podman logs  mitmproxy-default
echo ""
echo " Monitor the mitmproxy-default UI @ http://$(curl -s ifconfig.me):9081 "
echo "###############################################################################"
podman logs  mitmproxy-wireguard
echo ""
echo " Monitor the mitmproxy-wireguard UI @ http://$(curl -s ifconfig.me):8081 "
echo "###############################################################################"
echo ""
echo "Please update the mitmproxy wireguard client configuration endpoint address to:  $(curl -s ifconfig.me)"
echo ""
echo "###############################################################################"
echo ""

MITMproxy CERTS:

Add mitmproxy-ca-cert to the trusted root certs folder on your client host OS keystore (MS Win: certlm.msc) and/or if there is a java keystore for the client tool, please add the mitmproxy-ca-cert.cer as a trusted cert. keytool -import -trustcacerts -file mitm-ca-proxy.cer -alias mitmproxy -keystore capam.keystore

WireGuard client configuration:

To ensure that only selected web traffic is monitored through wireguard VPN to mitmproxy, make changes to the wireguard.conf file before importing it. Specifically, update the AllowedIPs address field to include a single IP address. Additionally, modify the endpoint field to direct traffic to the public IP address of the mitmproxy host on UDP port 51820. If deploying mitmproxy on AWS or other cloud hosts, confirm that the firewall/security groups permit TCP 8080, 8081, 9080, 9091, and UDP 51820. Once you have activated the WireGuard client, test your processes on the host and monitor the MITMproxy UI for updates.

An example of data captured between two (2) CLI tools. These CLI tools did not honor the OS environmental variables of HTTP_PROXY & HTTPS_PROXY. Using the MITMproxy with WireGuard process, we can now confirm the delta submission behavior that was masked by the CLI tools. This process was useful to confirm that MS Powershell was removing special characters for a password string, e.g. ! (exclamation mark).

Example of script deploying two (2) MITMproxy containers