Currently I have all my Debian virtual machines using a Squid caching server to cut down on bandwidth when they all go to pull updates and download packages. And all other server to internet HTTP traffic was decrypted passively by my firewall. I am looking at moving that decryption process off of the networking device and onto a dedicated proxy server.
I've already made it so HTTPs with apt is decrypted and inspected (I can see the URL path in an HTTPs connection from the logs), but using other programs on the server are not utilizing HTTP tunnels when the http_proxy or https_proxy variable is set.
To be clear, all programs on the servers are respecting the http(s)_proxy system variable, and are attempting to connect directly to Squid on Squid's port, but they are not utilizing HTTP tunnels (as in, they are not sending the prerequisite HTTP CONNECT packet prior to initializing their desired connection with Squid). I saw this behavior with both curl (without setting -x option), and crowdsec's cscli capi status, where both attempted to use the proxy but not did try to establish a tunnel with CONNECT.
Squid is then failing to proxy this direct TLS connection, as in the PCAPs it tries to respond with a HTTP packet after the client attempts TLS client-hello. The error log on Squid's side for this failed connection is fairly bland:
Jul 13 21:02:06 WebProxy squid[73457]: 1720929726.467 RELEASE -1 FFFFFFFF 0200000000000000F11E010001000000 400 1720929726 0 -1 text/html;charset=utf-8 3326/3326 NONE error:invalid-request Which is not surprising since Squid was expecting information over HTTP (like a CONNECT request), but instead got the first part of a TLS handshake.
My current squid config looks like this (I've omitted many acls for the sake of readability):
acl rproxy src 192.168.20.6 acl rproxy-d dstdomain "/etc/squid/mirror-dstdomain.acl.d/rproxy-domains.acl" # this contains the package blacklist acl blockedpkgs urlpath_regex "/etc/squid/pkg-blacklist-regexp.acl" # default to a different port than stock squid http_port 3142 ssl-bump generate-host-certificates=on dynamic_cert_mem_cache_size=4MB \ cert=/etc/squid/signingCA.crt \ key=/etc/squid/signingCA.key \ tls-cafile=/etc/squid/chain.pem sslcrtd_program /lib/squid/security_file_certgen -s /var/cache/squid/ssl_db -M 4MB sslcrtd_children 5 ssl_bump server-first all always_direct allow all forwarded_for delete # ------------------------------------------------- # user visible name visible_hostname webproxy # we need a big cache, some debs are huge maximum_object_size 512 MB # use a different dir than stock squid and default to 40G cache_dir aufs /var/spool/squid 40000 16 256 # custom log format logformat squid_access_detailed {"log":"squid-access","time":"%tl","responseTime":%tr,"srcIp":"%>a","srcPort":"%>p","destIp":"%<a","destPort":"%<p","userIdent":"%[ui","user":"%[un","method":"%rm","httpVer":"%>rv","url":"%ru","referrer":"%{Referer}>h","userAgent":"%{User-Agent}>h","status":"%>Hs","reqAction":"%Ss","reqStatus":"%Sh","contentType":"%mt","bytes":%st,"bytesIn":%>st,"bytesOut":%<st} # use different logs cache_access_log syslog:local4.info squid_access_detailed cache_log /var/log/squid/cache.log cache_store_log syslog:local6.info # tweaks to speed things up cache_mem 2048 MB maximum_object_size_in_memory 200240 KB # pid pid_filename /run/squid.pid # refresh pattern for debs and udebs refresh_pattern deb$ 129600 100% 129600 refresh_pattern udeb$ 129600 100% 129600 refresh_pattern tar.gz$ 129600 100% 129600 refresh_pattern tar.xz$ 129600 100% 129600 refresh_pattern tar.bz2$ 129600 100% 129600 # always refresh Packages and Release files refresh_pattern \/(Packages|Sources)(|\.bz2|\.gz|\.xz)$ 0 0% 0 refresh-ims refresh_pattern \/Release(|\.gpg)$ 0 0% 0 refresh-ims refresh_pattern \/InRelease$ 0 0% 0 refresh-ims refresh_pattern \/(Translation-.*)(|\.bz2|\.gz|\.xz)$ 0 0% 0 refresh-ims # handle meta-release and changelogs.ubuntu.com special # (fine to have this on debian too) refresh_pattern changelogs.ubuntu.com\/.* 0 1% 1 # only allow connects to ports for http, https acl Safe_ports port 80 acl Safe_ports port 443 # only allow ports we trust http_access deny !Safe_ports # do not allow to download from the pkg blacklist http_access deny blockedpkgs # Domain Allow by client IP http_access allow rproxy-d rproxy pinger_enable off # # And finally deny all other access to this proxy http_access deny all # we don't want to clash with the squid netdb state file netdb_filename stdio:/var/log/squid/netdb.state I am utilizing an intermediate CA to sign the fake server certificates so I don't have my network-wide CA private key sitting on this proxy server. The root CA cert is installed and trusted on all servers in the network.
Squid version information:
Squid Cache: Version 5.7 Service Name: squid Debian linux This binary uses OpenSSL 3.0.13 30 Jan 2024. configure options: '--build=x86_64-linux-gnu' '--prefix=/usr' '--includedir=${prefix}/include' '--mandir=${prefix}/share/man' '--infodir=${prefix}/share/info' '--sysconfdir=/etc' '--localstatedir=/var' '--disable-option-checking' '--disable-silent-rules' '--libdir=${prefix}/lib/x86_64-linux-gnu' '--runstatedir=/run' '--disable-maintainer-mode' '--disable-dependency-tracking' 'BUILDCXXFLAGS=-g -O2 -ffile-prefix-map=/build/reproducible-path/squid-5.7=. -fstack-protector-strong -Wformat -Werror=format-security -Wno-error=deprecated-declarations -Wdate-time -D_FORTIFY_SOURCE=2 -Wl,-z,relro -Wl,-z,now ' 'BUILDCXX=g++' '--with-build-environment=default' '--enable-build-info=Debian linux' '--datadir=/usr/share/squid' '--sysconfdir=/etc/squid' '--libexecdir=/usr/lib/squid' '--mandir=/usr/share/man' '--enable-inline' '--disable-arch-native' '--enable-async-io=8' '--enable-storeio=ufs,aufs,diskd,rock' '--enable-removal-policies=lru,heap' '--enable-delay-pools' '--enable-cache-digests' '--enable-icap-client' '--enable-follow-x-forwarded-for' '--enable-auth-basic=DB,fake,getpwnam,LDAP,NCSA,PAM,POP3,RADIUS,SASL,SMB' '--enable-auth-digest=file,LDAP' '--enable-auth-negotiate=kerberos,wrapper' '--enable-auth-ntlm=fake,SMB_LM' '--enable-external-acl-helpers=file_userip,kerberos_ldap_group,LDAP_group,session,SQL_session,time_quota,unix_group,wbinfo_group' '--enable-security-cert-validators=fake' '--enable-storeid-rewrite-helpers=file' '--enable-url-rewrite-helpers=fake' '--enable-eui' '--enable-esi' '--enable-icmp' '--enable-zph-qos' '--enable-ecap' '--disable-translation' '--with-swapdir=/var/spool/squid' '--with-logdir=/var/log/squid' '--with-pidfile=/run/squid.pid' '--with-filedescriptors=65536' '--with-large-files' '--with-default-user=proxy' '--enable-linux-netfilter' '--with-systemd' '--with-openssl' '--enable-ssl-crtd' 'build_alias=x86_64-linux-gnu' 'CFLAGS=-g -O2 -ffile-prefix-map=/build/reproducible-path/squid-5.7=. -fstack-protector-strong -Wformat -Werror=format-security -Wno-error=deprecated-declarations' 'LDFLAGS=-Wl,-z,relro -Wl,-z,now ' 'CPPFLAGS=-Wdate-time -D_FORTIFY_SOURCE=2' 'CXXFLAGS=-g -O2 -ffile-prefix-map=/build/reproducible-path/squid-5.7=. -fstack-protector-strong -Wformat -Werror=format-security -Wno-error=deprecated-declarations' I was hoping that maybe there was a way to define system-wide that all programs should use the Squid proxy via HTTP tunneling for their HTTPs needs?
If not, is there a way to make Squid intercept a direct HTTPs connection without a prior HTTP CONNECT? I attempted to set either the tproxy or intercept options on http_ports option in Squid, but I get No forward-proxy ports configured error on startup, but as I understand it, there is no need for forward-proxy ports as all clients are already sending their traffic directly to the Squid port.
