Routing fwmark to VPN gateway using nftables mark

I've learned a LOT from this question, but since nftables configuration is not "a piece of cake" to everyone, I want to share my own config, I'll try to share as detailed as I can so someone might have a more intuitive idea on how to do pbr wi nftables:

Step 1, define a standalone nft table.

 table inet classify { set pbrips { type ipv4_addr elements = { 1.0.0.1, 8.8.4.4, 8.8.8.8 } } chain output { type route hook output priority filter; policy accept; # This needs to be ^^^^^ route, not filter ip daddr @pbrips counter packets 29795 bytes 3021764 ct mark set 0x00000002 ip daddr @pbrips counter packets 29795 bytes 3021764 meta mark set ct mark } chain prerouting { type filter hook prerouting priority mangle; policy accept; ip daddr @pbrips counter packets 193712 bytes 60796012 ct mark set 0x00000002 ip daddr @pbrips counter packets 193712 bytes 60796012 meta mark set ct mark } } 

Here I defined a standalone table called classify, instead of using the existing fw4 table of openwrt 23.05, I do this because I tried to manipulate directly on fw4 table but it seemed to have no effect.

In the table I defined a nftset named pbrips, with some pre-defined ips (in my case I'll always re-route these ips). In this set, each ip is designed to route to another interface instead of the default gateway, thus it's called policy-based-routing, i.e. pbr.

You may have noticed the counter packets 29795 bytes 3021764 part, it's generated automatically by nftables, when you are creating rules, what you only need is a bare counter directive.

For some reasons I have to put two chains there, the output chain, and the prerouting chain, please pay attention to the hooks used inside each chain, they are the key-points if you want to re-route traffic, The reason I put two chains here is that If I only use the output chain, only the traffic (related to the 3 ips in the set) from the openwrt router itself can be re-routed, while with the prerouting chain all the traffic from my LAN can be correctly re-routed (p.s. If someone knows, please kindly remind me how to fix it Thks!)

Step 2, add a dedicated route table for pbr

Here is my /etc/iproute2/rt_tables:

root@OpenWrt:~# cat /etc/iproute2/rt_tables # # reserved values # 128 prelocal 255 local 254 main 253 default 0 unspec # # local # #1 inr.ruhep 123 pbr 

You can see in the last row I put a new table there, with id 123 and name pbr, you can use whatever id and name here, just don't use the already existing ones above, they are system-level values. you can edit this file directly with vi and save it

Step 3, create a route in the table to receive all traffic

Here is my output of ip route list table 123 (n.b. you can refer to the table either by it's id or it's name, they are both OK)

root@OpenWrt:~# ip route list table 123 default dev tun0 scope link root@OpenWrt:~# 

As you can see, there is only one rule in table 123, and if you want to re-route all traffic through table 123, this route is sufficient.

Step 4, the critical step, create ip rule to consume the fwmark

Here is my output of ip rule

root@OpenWrt:~# ip rule 0: from all lookup local 32765: from all fwmark 0x2 lookup pbr 32766: from all lookup main 32767: from all lookup default root@OpenWrt:~# 

And the keypoint here lies in the 3rd line, aka the fwmark 0x2 part, here 0x2 refers to hex value 2, but hex value 0x2 is equivalent to decimal value 2, and the value in my nft rule is 0x00000002, because 0x00000002 is exactly the same hex value of 0x2, and both equal to decimal value 2, I recommend you to always use a decimal value like 1,2,3, not 0x1,0x2,0x3, to not confuse yourself (as I did before)

Step 5, use whatever your preferred method to add elements to the nftset

As long as you add new elements (i.e. IPs) to the nftset pbrips, you can test it out through traceroute 8.8.8.8 to verify if the pbr policy is working or not. Theoretically you are ready to go now.

Step 6, you may need to change the value of sysctl rp_filter

As mentioned on this Medium blog post you may need to change the value to echo 2 > /proc/sys/net/ipv4/conf/all/rp_filter or echo 0 > /proc/sys/net/ipv4/conf/all/rp_filter (risky, check the documentation).

echo "net.ipv4.conf.all.rp_filter=2" > /etc/sysctl.d/95-IPv4-Filtering.conf sysctl -p /etc/sysctl.d/95-IPv4-Filtering.conf # Reboot and check the value sysctl -a | grep rp_filter 

And Thank you again Alexey Martemyanov for your question and your answer as a good reference to help me learning pbr!

Answering my own question in case it could be useful for someone:

After finding "[SOLVED] iproute2 ignores connection marks set with nftables" thread I've updated the nft rules set to the following:

nft insert rule inet fw4 mangle_prerouting ip daddr @marker counter ct mark set 390 nft add rule inet fw4 mangle_prerouting ip daddr @marker counter meta mark set ct mark 

And it worked! Turns out I needed to set a connection mark not a meta mark