OpenBSD routing: Cannot reach gateway from IF configured to static block

Question

When trying to set up an OpenBSD router I've run into an apparent routing problem.

I have a 1U machine with 6 gigabit NICs (em0-em5). My ISP provided me with the following:

xx.xx.97.246/28 static WAN IP. xx.xx.97.241 default gateway address. xx.xx.98.192/29 static block that is externally routed to via xx.xx.97.246 

This is my setup:

+-------------------+ | em0 +----> ISP's gateway (xx.xx.97.241) | xx.xx.97.246 | +-------------------+ | em2 +----> DMZ network (xx.xx.98.194-198) | xx.xx.98.193/29 | +-------------------+ | em4 +----> LAN2 (172.16.1.0/24) | 172.16.1.1 | [NATs to xx.xx.97.246 address] +-------------------+ | em5 +----> LAN (192.168.1.0/24) | 192.168.1.1 | [NATs to xx.xx.97.246 address] +-------------------+ 

Forwarding has been enabled in sysctl.conf.

With this setup and some basic PF rules I have been able to get basic Internet access from the LAN networks (em4/5) and I can access mail/DNS servers on the DMZ network from the LAN networks. So far, so good.

The problem is in accessing the DMZ network from the Internet and any host on the DMZ network accessing the Internet (to perform upstream DNS look ups, etc). There does not seem to be any connectivity to the Internet in either direction when a DMZ net address is the endpoint.

I can ping the xx.xx.97.246 WAN address from the DMZ net but I cannot ping the ISP's default gateway address (xx.xx.97.241).

I get an odd trace when trying to traceroute from DMZ net to the WAN IP - it skips the em2 hop and only reports the WAN IP as if it was the next hop. I can trace to em2 and get it's address as the next hop if I specify the em2 address as the destination address. Regardless I cannot ping nor trace to the gateway address - I get a "host unreachable" message.

This is what I have tried so far:

1. Adding a static route from xx.xx.98.193 to the WAN IP and got "route exists" ok, makes sense. Out of desperation I tried adding a route from xx.xx.98.193 to the gateway address and got "destination unreachable".

2. Adding xx.xx.98.193 as an IP alias to em0 and changing the address of em2 to xx.xx.98.194. This did not seem to have any effect on anything besides breaking all routing through the gateway.

3. Unplugged the DMZ switch from em2 and plugged in a PC configured to static address xx.xx.98.194 then disabled pf with pfctl -d which essentially passes everything and takes pf out of the picture. No change.

4. Double-checked the arp table with arp -a and can see that em2 (it's MAC) is bound to xx.xx.98.193/29. Also checked the routing table with netstat -ranf inet and it shows the xx.xx.97.241 gateway as the default.

At this point I think this is purely a routing problem especially after taking steps 3 & 4. By the way, steps 1 & 2 were reversed and the system rebooted to revert back to the previous config before trying 3 & 4.

I've considered bridging em0 and em2 but it just doesn't seem necessary since all the interfaces are in the same machine and once an em2 host's packet makes it into the router the routing table should have the path out to the default gateway.

I also noticed an odd arp message repeating in /var/log/messages "attempt to overwrite permanent entry xx.xx.98.193 by ". The mac via arp was from the ISP's gateway xx.xx.97.241. Not sure if this is normal "chatty logging" or if it is a symptom of my problem.

Thanks in advance for your time reading and responding,

Rob

UPDATE:

Here is my routing table:

Routing tables Internet: Destination Gateway Flags Refs Use Mtu Prio Iface default xx.xx.97.241 UGS 4 54 - 8 em0 127/8 127.0.0.1 UGRS 0 0 32768 8 lo0 127.0.0.1 127.0.0.1 UHl 1 0 32768 1 lo0 172.16.1/24 172.16.1.1 C 0 0 - 8 em4 172.16.1.1 xx:xx:xx:xx:1b:60 HLl 0 0 - 1 lo0 172.16.1.255 172.16.1.1 Hb 0 0 - 1 em4 192.168.1/24 192.168.1.1 C 0 0 - 8 em5 192.168.1.1 xx:xx:xx:xx:1b:61 HLl 0 0 - 1 lo0 192.168.1.255 192.168.1.1 Hb 0 0 - 1 em5 xx.xx.97.240/28 xx.xx.97.246 UC 1 0 - 8 em0 xx.xx.97.241 xx:xx:xx:xx:fc:d4 UHLc 1 4 - 8 em0 xx.xx.97.246 xx:xx:xx:xx:1b:5c HLl 0 0 - 1 lo0 xx.xx.97.255 xx.xx.97.246 UHb 0 0 - 1 em0 xx.xx.98.192/29 xx.xx.98.193 UC 2 0 - 8 em2 xx.xx.98.193 xx:xx:xx:xx:1b:5e HLl 0 0 - 1 lo0 xx.xx.98.195 xx:xx:xx:xx:d8:33 UHLc 0 4 - 8 em2 xx.xx.98.197 link#3 UHLc 0 4 - 8 em2 xx.xx.98.199 xx.xx.98.193 UHb 0 0 - 1 em2 224/4 127.0.0.1 URS 0 0 32768 8 lo0 

I tried a lot of configs last night (primarily because I'm running out of ideas). As per the previous attempts, PF was disabled with pfctl -d to remove it as a variable. Probably the most sane attempt at something was this:

1. Unplugged em4 and em5 (the dynamic nets) and removed their hostname.if files from /etc to leave only em0 and em2.
2. Configured em0 with the xx.xx.98.193 address and em2 with the xx.xx.98.194 address then a PC connected to em2 with xx.xx.98.195.

With that configuration from the PC I could ping up to the 194 address (em2 IF) but not the 193 address (em0 IF). From the OpenBSD system I could ping the 194 and 195 addresses.

It seems like whatever the problem is the breakdown in routing is occurring when em2 tries to reach em0. Also for at least one ping attempt the PC made it up to em0 with 50 loss but consistently failed thereafter.

There must be something silly that I am missing here. My next crazy thing to try will be to enable/add kernel routing logs and compare what is output between successful routing between interfaces and between em0/2. Its going to be a crash course in OpenBSD's networking stack/kernel for that. :/

Answer 1

Post the output of netstat -ranf inet. What happens if you disable pf and ping a host on the Internet that’s known to respond using ping -I xx.xx.98.193? I also find your ISP providing a MAC address for xx.xx.98.193 unusual—it leads me to believe your problem is on your ISP’s end.