Today we allow compressing transport traffic via transport.tcp.compress. This is all or nothing, it either applies to every transport connection or to none. This means that we can not enable compression on remote-cluster traffic independently from within-cluster traffic. There are compelling reasons to allow these to be separate though. We expect that some architectures will have fast within-cluster interconnects (e.g., 10 GbE or more) but not-so-fast remote cluster interconnects. In these cases, trading CPU for compression on the within-cluster traffic could be a waste, but critical for the remote-cluster traffic.

This is especially important for cross-cluster replication where we expect large volumes of data to be transported across potentially small tubes.

We should enable setting transport.tcp.compress on a per-connection basis so that it can be separate for within-cluster traffic versus each remote-cluster connection.

It is worth considering if there are other transport-level settings that would benefit from this enablement as well.