Set HTTP/HTTPS man-in-the-middle proxy for webserver (Apache2)

To log all http/https requests by a specific application, we use man-in-the-middle proxies (set as http or SOCKS5 proxies). On our local Windows and Mac machines, we usually 'proxify' apps or temporarily set a system-wide proxy. This works like a charm for any app we can think of. On our remote Linux boxes, truly system-wide proxy are causing connection problems (and it's also somewhat harder to achieve the same results in a non-desktop environment). So we decided to only go with proxifying seperate applications there.

To do this, we currently use mitmproxy + tsocks/proxychains. This once again works great for normal applications. The current problem we have however, is that we can't seem to get it working with our apache2/php services. We have been looking for solutions for the past few days, but it's hard to find options that are not about turning the webserver itself in to a reverse proxy from or to a specific url. What we want is to monitor any remote http/https request the webserver does (outbound requests of a PHP application). We thought of running the webserver in Docker and set a proxy for the container, but that means we have to set everything up again in the container and it feels like a bad solution that comes with its own troubles.

The actual question: how can we route all http/https traffic our webserver does through our MITM proxy, so we can see all requests to remote http/https hosts?

Alternative/better solutions to achieve the same are welcome.

Thanks to Bob for pointing me in the right direction. Sometimes things are truly just a Google search away once you know what to look for.

@Bob commented:

A fairly conventional way to run a proxy for applications that don't honor proxy settings is to use firewall rules to rewrite all outgoing traffic with a web port as the destination to a transparent proxy. Fairly trivial for plain http, but slightly more involved when it concerns https.

Instead of setting the proxy in the proxy settings of the OS, we can configure iptables to route everything on port 80 and 443 to our proxy (steps taken from docs.mitmproxy.org)

Enable IP forwarding and disable ICMP redirects:

sudo sysctl -w net.ipv4.ip_forward=1
sudo sysctl -w net.ipv6.conf.all.forwarding=1
sudo sysctl -w net.ipv4.conf.all.send_redirects=0

Skip step 3 and 4 of the manual, because we want to redirect traffic originating from the machine itself.

To do this, create a user to run mitmproxy to route traffic to:

sudo useradd --create-home mitmproxyuser
sudo -u mitmproxyuser -H bash -c 'cd ~ && pip install --user mitmproxy'

And finally the iptables rules (proxy running on port 8080):

sudo iptables -t nat -A OUTPUT -p tcp -m owner ! --uid-owner mitmproxyuser --dport 80 -j REDIRECT --to-port 8080
sudo iptables -t nat -A OUTPUT -p tcp -m owner ! --uid-owner mitmproxyuser --dport 443 -j REDIRECT --to-port 8080
sudo ip6tables -t nat -A OUTPUT -p tcp -m owner ! --uid-owner mitmproxyuser --dport 80 -j REDIRECT --to-port 8080
sudo ip6tables -t nat -A OUTPUT -p tcp -m owner ! --uid-owner mitmproxyuser --dport 443 -j REDIRECT --to-port 8080

Finally, run mitmproxy as the new user:

sudo -u mitmproxyuser -H bash -c 'mitmproxy --mode transparent --showhost --set block_global=false'

That's as far as the manual provided by mitmproxy goes. At this point, we're done with the setup, but we need to install a new certificate (any previous ones from mitm.it will not work).

After the first run, a new certificate is generated at /home/mitmproxyuser/.mitmproxy/mitmproxy-ca-cert.pem. We need to convert that to crt and install it:

sudo openssl x509 -in /home/mitmproxyuser/.mitmproxy/mitmproxy-ca-cert.pem -inform PEM -out /home/mitmproxyuser/.mitmproxy/mitmproxy-ca-cert.crt
sudo cp /home/mitmproxyuser/.mitmproxy/mitmproxy-ca-cert.crt /usr/share/ca-certificates/extra/mitmproxy-ca-cert.crt
sudo dpkg-reconfigure ca-certificates

At this point, you're almost set, except that Apache2/PHP still won't trust our proxy certificate so external requests won't get through. I tried a bunch of apache.conf SSL configurations that didn't work. Turned out, I had to uncomment the following line in /etc/php/7.4/apache2/php.ini and link it to the crt file we created:

openssl.cafile=/home/mitmproxyuser/.mitmproxy/mitmproxy-ca-cert.crt

Don't forget to restart apache2/php7.4-fpm and that's it! Our PHP app is now routed through the proxy. You may also need to trust the generated crt file in your browser to get rid of the usual warning.