Dual IP Source Based Routing on Server

Question

I have struggled in an enterprise environment with IP changes (moving VLANS) of servers. Temporary dual homing would be an ideal solution, but after trying on and off for a year, I have always had routing issues from systems within the same VLAN.

We use Ubuntu with Netplan, below is an example Netplan configuration. Everything works as expected, except traffic within the local VLAN for the new VLAN (20 in this example).

Routing between the VLANs happens on a core switch, both server interfaces are on a Cisco switch. Traffic between VLANs also is routed through a firewall but the rules between these two are any:any.

Additional Testing:

Using tcpdump and ping, it appears inbound traffic from VLAN 10 (using the interface IP of VLAN 20) exits on VLAN 20 and is lost. Basically it is not routing traffic back on the source interface when it is coming from VLAN 20 into VLAN 10s interface from the enterprise network side.

I am not sure how to fix that. How does one ensure a true source-based routing in this case is it even possible?

Netplan configuration:

# VLAN 10 (eno2np1) (10.86.152.0/22) # VLAN 20 (eno4) (10.126.110.1/26) # Traffic is lost within VLAN 20. tcpdump sees packets on local and remote systems, unclear exactly what is causing lack of communication. network: ethernets: # Primary Server Address / Adapter. # VLAN 10 eno2np1: dhcp4: false addresses: [10.86.152.220/22] routes: - to: default via: 10.86.152.1 routing-policy: - from: 10.86.152.220 nameservers: addresses: [10.56.56.56,10.56.56.156] # New VLAN we are moving to. # VLAN 20 eno4: dhcp4: false addresses: [10.126.110.20/26] nameservers: addresses: [10.56.56.56,10.56.56.156] routes: - to: default via: 10.126.110.1 table: 100 routing-policy: - from: 10.126.110.20 table: 100 version: 2 

/etc/iproute2/rt_tables:

# # reserved values # 255 local 254 main 253 default 0 unspec # # local # #1 inr.ruhep # 100 ipten 

ip tool results:

Answer 1

What you described is source-based routing, only that term really means source-address-based routing. If you send packets from address Y, you're using the routing table Y, and so on.

What you're looking for is, I think, might be called flow-based routing. To implement that, use iptables rules to set a connmark for all inbound packets, to restore that connmark to the packet-level mark for outbound packets, and change your policy-routing rules to match on the fwmark instead of the source address.

Answer 2

Working solution with Netplan config. Hope this helps someone else.

So after a painful amount of reading, I realized that the main route table had routes for local links related to both adapter IPs.

I discovered how to drop the route from the main, and add the on-link route to the ipten table. Suddenly traffic from with-in the VLAN started to flow as expected.

I then spent time trying to figure out how to properly configure Netplan to auto create the needed routes. Adding the on-link route to the 2nd route table was easy:

 - to: 10.126.110.0/26 via: 10.126.110.38 on-link: true table: ipten 

What I could not figure out was how to prevent the main route table from getting both link routes. I could manually drop the highlighted route above, but if you wanted this configuration long term, I needed to get Netplan correct. After additional fighting, I found it easier to just make an additional route table in /etc/iproute2/rt_tables

# Custom tables added to /etc/iproute2/rt_tables 100 aux 200 prim 

You then need to add rules for when those tables are used, and to delete the rule for the main table from your local IP, main will still be used as a last resort since it is repeated at the bottom of the list.

sudo ip rule add from 10.86.152.218 table prim sudo ip rule add from 10.126.110.38 table aux sudo ip rule delete from 10.86.152.218 table main 

Your table should look like this:

I then setup a Netplan config to use both tables (aux and prim), and no main table.

network: version: 2 ethernets: #VLAN10 (Route table: prim) enp0s3: dhcp4: no link-local: [ ipv4 ] addresses: [10.86.152.218/22] routes: - to: default via: 10.86.152.1 table: 200 - to: 10.86.152.0/22 via: 10.86.152.218 scope: link table: 200 routing-policy: - from: 10.86.152.218 table: 200 nameservers: addresses: [10.56.56.56] #VLAN20 (Route table: aux) enp0s8: dhcp4: no link-local: [ ipv4 ] addresses: [10.126.110.38/26] routes: - to: default via: 10.126.110.1 table: 100 - to: 10.126.110.0/26 via: 10.126.110.38 scope: link table: 100 - to: default #add to main table, as a catch all for local outbound. via: 10.126.110.1 routing-policy: - from: 10.126.110.38 table: 100 nameservers: addresses: [10.56.56.56] 

Resulting route tables: