I have a db server (let's call it DB) on another cloud service and a VPN server running wireguard on Amazon AWS (let's call it GW), an EC2 instance. I also have a web server as an EC2 instance (let's call it WEB).
I'm a complete noob to AWS services. My networking setup contains the following:
- A VPC containing two subnets, one public (let's call it PUB), one private (let's call it PVT).
- An internet gateway at the PUB subnet
- An Elastic IP attached to one of GW's network interface
The GW instance has two network interfaces:
- one at the PUB subnet (
10.25.0.2/24) with the EIP attributed - one at the PVT subnet (
10.25.240.2/24)
The WEB instance has one network interface (10.25.240.50/24).
Both have private IPv4, only the GW has a public IPv4, and both have IPv6, but I'm focusing on setting up the IPv4 first, so let's ignore the IPv6 setup.
There's a Wireguard tunnel established between DB and GW with the following setup:
- GW:
wg0,192.168.40.1/24 - DB:
wg0,192.168.40.2/24
Both DB and GW ping each other through the tunnel, and both GW and WEB ping each other through private subnet interfaces. I did a "allow everything" Security Group for both instances at the interface that communicate with each other because I suspected it could be a problem.
All instances run Linux and GW has sys.net.ipv4.ip_forward sysctl option set to 1.
I tried disabling my firewall (firewalld), I tried creating policies for inter-zone traffic forward, I tried everything, but packets from DB simply won't arrive at WEB (they do leave GW, though) and packets from WEB simply won't arrive at GW.
I tested with ICMP packets running tcpdump, packets destined to WEB from DB arrive at the tunnel interface, and they are sent to the wire into the private subnet from dumping the private subnet's interface, but tcpdump at the WEB instance doesn't show anything arriving). Also, packets from the WEB destined to DB are captured at the WEB network interface, but won't appear at the GW interface at all.
DB routing table:
default via 10.1.1.1 dev eth0 proto dhcp src 10.1.1.149 metric 100 10.1.1.0/24 dev eth0 proto kernel scope link src 10.1.1.149 metric 100 10.25.240.0/24 dev wg0 scope link 192.168.40.0/24 dev wg0 proto kernel scope link src 192.168.40.2 (the route to 10.25.240.0/24 was produced by wireguard's AllowedIPs)
WEB Routing Table
default via 10.25.240.1 dev eth0 proto dhcp src 10.25.240.50 metric 100 10.25.240.0/24 dev eth0 proto kernel scope link src 10.25.240.50 metric 100 192.168.40.0/24 via 10.25.240.2 dev eth0 (the route to 192.168.40.0/24 was manually added to NetworkManager config)
That said, I have a twofold question:
- In general, how would I approach this sort of situation to diagnose the issue when working with AWS stuff?
- In specific, what could be the possible cause and the possible solutions for this issue?
PS: I am somewhat aware AWS has services that would replace the need for some of the elements on my setup, but I would rather use this setup for better integration with already existing solutions.
