2

Quick overview of the situation

I have a Fedora host on which there are multiple NICs and VMs (QEMU/KVM/LibVirt). Each guest accesses the internet via a virtual bridge that operates in NAT mode and forwards traffic to a single physical device.

For reasons that are beyond me, when it comes to this USB/Ethernet adapter (an Android phone), I can't seem to make it work as I do with other NICs. The guest receives DNS answers from the host, then tries to access the internet, and gets nothing but ICMP host unreachable and TTL expired types of responses.

A description of how my infrastructure works

Here's a rundown of how it usually works. The problematic guest/bridge/NIC trio will come later (but is supposed to work in much the same way).

The host

An x86_64 Fedora 40 workstation.

➜ ~ uname -r 6.10.6-200.fc40.x86_64

The hypervisor

QEMU/KVM, LibVirt, using virt-manager or virsh depending on needs.

➜ ~ qemu-kvm --version QEMU emulator version 8.2.6 (qemu-8.2.6-3.fc40) Copyright (c) 2003-2023 Fabrice Bellard and the QEMU Project developers 
➜ ~ virsh --version=long Virsh command line tool of libvirt 10.1.0 See web site at https://libvirt.org/ Compiled with support for: Hypervisors: QEMU/KVM LXC LibXL OpenVZ VMware VirtualBox ESX Hyper-V Test Networking: Remote Network Bridging Interface udev Nwfilter Storage: Dir Disk Filesystem SCSI Multipath iSCSI iSCSI-direct LVM RBD Gluster ZFS Miscellaneous: Daemon Nodedev SELinux Secrets Debug DTrace Readline 
➜ ~ virt-manager --version 4.1.0

The guest

Clean Win10 install, ISO from MS, winFSP and Virtio-win-guest-tools installed.

C:\Users\User>ver Microsoft Windows [version 10.0.19045.4780] 
PS C:\Windows\system32> winget list Nom ID Version Disponible Source ----------------------------------------------------------------------------------------------------------------------- WinFsp 2023 WinFsp.WinFsp 2.0.23075 winget Virtio-win-guest-tools ARP\Machine\X86\{c5ad265e-98e6-40bd-801… 0.1.262

Overview of the usual setup

In summary

Here is how I usually do things. The host is connected to a router via a physical Ethernet NIC, let's call it eth0 on my_network. A virtual network is created in virt-manager, let's call it my_virt_net. It has a virtual bridge that operates in NAT mode and forwards to physical device eth0, let's call it virbr0. The guest gets a virtual network interface corresponding to that network source, set up as a virtIO device, let's call it vNIC0.

The guest's vNIC gets assigned an IP via DHCP (libvirt runs a DNSMASQ instance for each, I think ?). In this example, we are operating on 10.0.0.0/24, the gateway being 10.0.0.1.

IP rules and routes are created to make sure the traffic of the guest goes through the right device. Libvirt takes care of creating MASQUERADE rules in the iptables.

Once all this is done, the guest has access to the internet. Traffic flows the following way:

Guest <-> vNIC0 <-> virbr0 <-> vnet0 <-> eth0 <-> my_network <-> outside_world

In detail

➜ ~ nmcli con show NAME UUID TYPE DEVICE my_network 3d72b049-b1a7-49e8-ab55-18edf0d79de9 ethernet enp0s31f6 
➜ ~ ip addr 1: enp0s31f6: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000 link/ether 40:8d:5c:4d:e5:c0 brd ff:ff:ff:ff:ff:ff inet 192.168.1.1/24 brd 192.168.1.255 scope global dynamic noprefixroute enp0s31f6 valid_lft 40431sec preferred_lft 40431sec inet6 2a01:e0a:18b:b670:3429:7bc3:6997:2e17/64 scope global dynamic noprefixroute valid_lft 86265sec preferred_lft 86265sec inet6 fe80::ca7c:24c:ce13:2a22/64 scope link noprefixroute valid_lft forever preferred_lft forever 2: virbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000 link/ether 52:54:00:ed:b3:89 brd ff:ff:ff:ff:ff:ff inet 10.0.0.1/24 brd 10.0.0.255 scope global virbr0 valid_lft forever preferred_lft forever 3: vnet0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master virbr0 state UNKNOWN group default qlen 1000 link/ether fe:54:00:32:2f:3e brd ff:ff:ff:ff:ff:ff inet6 fe80::fc54:ff:fe32:2f3e/64 scope link proto kernel_ll valid_lft forever preferred_lft forever 
➜ ~ brctl show bridge name bridge id STP enabled interfaces virbr0 8000.525400edb389 yes vnet0 
➜ ~ virsh net-list Name State Autostart Persistent ----------------------------------------------------- my_virt_net active yes yes 
➜ ~ ip rule 0: from all lookup local 100: from all iif virbr0 lookup 100 32766: from all lookup main 32767: from all lookup default 
➜ ~ ip route show table 100 default dev enp0s31f6 scope link 
➜ ~ ip route default via 192.168.1.254 dev enp0s31f6 proto dhcp src 192.168.1.1 metric 100 10.0.0.0/24 dev virbr0 proto kernel scope link src 10.0.0.1 192.168.1.0/24 dev enp0s31f6 proto kernel scope link src 192.168.1.1 metric 100 
➜ ~ iptables -t nat -L -v -n Chain PREROUTING (policy ACCEPT 0 packets, 0 bytes) pkts bytes target prot opt in out source destination Chain INPUT (policy ACCEPT 0 packets, 0 bytes) pkts bytes target prot opt in out source destination Chain OUTPUT (policy ACCEPT 0 packets, 0 bytes) pkts bytes target prot opt in out source destination Chain POSTROUTING (policy ACCEPT 19151 packets, 2171K bytes) pkts bytes target prot opt in out source destination 21657 2313K LIBVIRT_PRT 0 -- * * 0.0.0.0/0 0.0.0.0/0 Chain LIBVIRT_PRT (1 references) 0 0 RETURN 0 -- * enp0s31f6 10.0.0.0/24 224.0.0.0/24 0 0 RETURN 0 -- * enp0s31f6 10.0.0.0/24 255.255.255.255 19 988 MASQUERADE 6 -- * enp0s31f6 10.0.0.0/24 !10.0.0.0/24 masq ports: 1024-65535 4 5112 MASQUERADE 17 -- * enp0s31f6 10.0.0.0/24 !10.0.0.0/24 masq ports: 1024-65535 0 0 MASQUERADE 0 -- * enp0s31f6 10.0.0.0/24 !10.0.0.0/24

This is a pretty standard setup I think. It works well for my needs. The outputs I have shown are simplified since I have, in fact, multiple guests and NICs, but you get the idea. It's always the same setup, and it works.

One thing to note is that I use scripts placed in /etc/NetworkManager/dispatcher.d/ to add or remove ip rules and routes when interfaces come up or down.

So what's the problem ?

Here's what's happening. I want to use a USB/Ethernet device (an android phone) tethered to the host, and set ip up like any other NIC. I want that phone to be used as a gateway to the outside world, by one and only one guest, and the host should preferably not be able to use it. My usual setup is to have everything well isolated.

The phone is connected to the host and is configured to automatically turn on USB tethering. It is detected as a network interface, and provides a DHCP lease, as expected:

➜ ~ ip addr 8: enp0s20f0u6: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UNKNOWN group default qlen 1000 link/ether 46:7a:42:be:dc:85 brd ff:ff:ff:ff:ff:ff inet 192.168.171.52/24 brd 192.168.171.255 scope global dynamic noprefixroute enp0s20f0u6 valid_lft 2302sec preferred_lft 2302sec inet6 2a02:8440:6103:24e4:d1fa:3bae:2f95:5b9d/64 scope global deprecated dynamic noprefixroute valid_lft 1503sec preferred_lft 0sec inet6 2a02:8440:6315:5cc:8eb1:665a:abb2:6847/64 scope global dynamic noprefixroute valid_lft 7108sec preferred_lft 7108sec inet6 fe80::faaa:5745:cf53:55cf/64 scope link noprefixroute valid_lft forever preferred_lft forever 13: virbr3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000 link/ether 52:54:00:eb:85:6a brd ff:ff:ff:ff:ff:ff inet 10.0.3.1/24 brd 10.0.3.255 scope global virbr3 valid_lft forever preferred_lft forever

Making sure we have connectivity to the outside world:

➜ ~ traceroute -i enp0s20f0u6 www.google.com traceroute to www.google.com (142.250.178.132), 30 hops max, 60 byte packets 1 _gateway (192.168.171.21) 0.831 ms 0.711 ms 0.803 ms ... ... ... [REDACTED] ... ... ... 15 par21s22-in-f4.1e100.net (142.250.178.132) 49.062 ms 66.086 ms 65.975 ms

So I do the usual. I create a virtual network in virt-manager (NAT mode, forward to physical device enp0s20f0u6 specifically) and attach it as a virtIO device to the guest. I set up ip rules and routes in a script that runs when the interface comes up or down:

➜ ~ sudo nano /etc/NetworkManager/dispatcher.d/103-motoE14-handler.sh 
#!/bin/bash interface=$1 event=$2 if [ "$interface" = "enp0s20f0u6" ]; then case "$event" in up) ip rule add prio 103 iif virbr3 table 103 ip route add default dev "$interface" table 103 ;; down) ip rule del iif virbr3 table 103 ip route del table 103 default dev "$interface" ;; esac fi 
➜ ~ sudo chmod +x /etc/NetworkManager/dispatcher.d/103-motoE14-handler.sh

Although this is exactly the same setup as for other guests and NICs, the guest has no connectivity to the outside world.

Sanity checks

Just to be clear, I did the following, and more, before posting:

  1. Try another phone: no bueno.
  2. Try another guest: no bueno.
  3. Nuke the config, rebuild from scratch: no bueno.
  4. Try another NIC on the problematic guest: it works.
  5. Try the problematic NIC on the host: it works.
  6. Try with all other NICs down and only this one remaining, including setting it as the default gateway for all traffic: no change.
  7. n-tuple check the ip rules, ip routes, ip tables: looks good to me.
  8. Check the firewall. All looks good to me.
  9. Try a new guest running Fedora 40: same issue.
  10. Use USB passthrough: it works, but I would like to manage all my interfaces in the same way.
  11. Use an e1000e device model instead of virtIO: no bueno.

Network traffic capture

I'm trying to read through TCP dumps that I capture from the host, on the physical interface, on the bridge, and on the vnet, but this is beyond my understanding of networking...

Note the following:

  1. "Fidel" is the name of the host.
  2. The USB/Ethernet NIC, a.k.a Android Phone, is:
  • enp0s20f0u6 (interface name).
  • 46:7a:42:be:dc:85 (MAC address).
  • 192.168.171.21 (Gateway IP).
  1. 192.168.171.52 is the IPv4 leased by the NIC to the host via DHCP.
  2. The virtual bridge is:
  • virbr3.
  • 52:54:00:eb:85:6a.
  • 10.0.3.1.
  1. The virtual NIC attached to the guest is:
  • 52:54:00:43:16:2f
  • 10.0.3.219

tcpdump from host: enp0s20f0u6

➜ ~ sudo tcpdump -e -E -f -i enp0s20f0u6 dropped privs to tcpdump tcpdump: verbose output suppressed, use -v[v]... for full protocol decode listening on enp0s20f0u6, link-type EN10MB (Ethernet), snapshot length 262144 bytes 18:48:37.824751 46:7a:42:be:dc:85 (oui Unknown) > a6:fe:e6:ee:73:2d (oui Unknown), ethertype IPv6 (0x86dd), length 106: 2a02-8440-6315-05cc-8eb1-665a-abb2-6847.rev.sfr.net.57013 > 2a02-8440-6315-05cc-0000-0000-0000-0033.rev.sfr.net.domain: 3959+ A? checkappexec.microsoft.com. (44) 18:48:37.838107 46:7a:42:be:dc:85 (oui Unknown) > Broadcast, ethertype ARP (0x0806), length 42: Request who-has 20.191.45.158 tell fidel, length 28 18:48:37.867576 46:7a:42:be:dc:85 (oui Unknown) > a6:fe:e6:ee:73:2d (oui Unknown), ethertype IPv6 (0x86dd), length 152: 2a02-8440-6315-05cc-8eb1-665a-abb2-6847.rev.sfr.net.57276 > 2a02-8440-6315-05cc-0000-0000-0000-0033.rev.sfr.net.domain: 10318+ PTR? 3.3.0.0.0.0.0.0.0.0.0.0.0.0.0.0.c.c.5.0.5.1.3.6.0.4.4.8.2.0.a.2.ip6.arpa. (90) 18:48:37.988238 a6:fe:e6:ee:73:2d (oui Unknown) > 46:7a:42:be:dc:85 (oui Unknown), ethertype IPv6 (0x86dd), length 232: 2a02-8440-6315-05cc-0000-0000-0000-0033.rev.sfr.net.domain > 2a02-8440-6315-05cc-8eb1-665a-abb2-6847.rev.sfr.net.57013: 3959 3/0/0 CNAME prod-atm-wds-apprep.trafficmanager.net., CNAME prod-agic-we-3.westeurope.cloudapp.azure.com., A 20.56.187.20 (170)

tcpdump from host: virbr3

➜ ~ sudo tcpdump -e -E -f -i virbr3 dropped privs to tcpdump tcpdump: verbose output suppressed, use -v[v]... for full protocol decode listening on virbr3, link-type EN10MB (Ethernet), snapshot length 262144 bytes 19:14:19.404024 52:54:00:43:16:2f (oui Unknown) > 52:54:00:eb:85:6a (oui Unknown), ethertype IPv4 (0x0800), length 74: 10.0.3.219.62464 > fidel.domain: 54546+ A? www.google.com. (32) 19:14:19.414360 52:54:00:eb:85:6a (oui Unknown) > 52:54:00:43:16:2f (oui Unknown), ethertype IPv4 (0x0800), length 90: fidel.domain > 10.0.3.219.62464: 54546 1/0/0 A 216.58.213.68 (48) 19:14:19.427738 52:54:00:43:16:2f (oui Unknown) > 52:54:00:eb:85:6a (oui Unknown), ethertype IPv4 (0x0800), length 106: 10.0.3.219 > par21s18-in-f4.1e100.net: ICMP echo request, id 1, seq 13, length 72 19:14:19.427786 52:54:00:eb:85:6a (oui Unknown) > 52:54:00:43:16:2f (oui Unknown), ethertype IPv4 (0x0800), length 134: fidel > 10.0.3.219: ICMP time exceeded in-transit, length 100 19:14:19.429008 52:54:00:43:16:2f (oui Unknown) > 52:54:00:eb:85:6a (oui Unknown), ethertype IPv4 (0x0800), length 106: 10.0.3.219 > par21s18-in-f4.1e100.net: ICMP echo request, id 1, seq 14, length 72

What do I see ?

When the guest asks for a DNS resolution, the host seems to provide that (and indeed, the traceroute output in the windows guest shows that the IP address of www.google.com has been found, but then the route can't be... traced).

Does anyone have suggestions as to what's happening ?

Edit: I've had to shorten the outputs of tcpdump because StackExchange was detecting my post as spam... I will provide more in subsequent posts if needed.

Improve this question

1 Answer 1

Reset to default
0

ip route add default dev "$interface" is, in most cases, an incomplete default route. When you don't specify a gateway IP address, it declares all of 0.0.0.0/0 to be "on link" or "local subnet" – instead of the OS making an ARP query for the gateway, it's told to make ARP queries for every destination IP directly, just as if they all were on your subnet. (Hence the Request who-has 20.191.45.158 ARP query in your tcpdump.)

This only works if your gateways provide Proxy-ARP and respond to ARP on behalf of distant addresses – which is something that AFAIK only Cisco enables out of the box for historical reasons (it's a feature that predates even the concept of subnetting). Most other routers, even if capable of doing Proxy-ARP, don't provide it by default; and I'm pretty sure that Android also does not.

The lack of ARP response is what leads to the ICMP error "Host unreachable" (not to be confused with "Network unreachable").

Because you cannot enable Proxy-ARP on standard Android1, you'll have to define the default route with the gateway address specified. (Since you probably have DHCP to get the IP address from Android, it'd be best to make the DHCP client directly add the route to the correct table.)

1 Unless the Android device is rooted, in which case regular Linux sysctls should work: net.ipv4.conf.xxx.proxy_arp would make that interface respond to ARP on behalf of any IP address that's routed through a different interface.

Even if Proxy-ARP is available (such as for your main wired connections), a gateway route is still preferable over a route that relies on Proxy-ARP, as in the latter case your host's ARP cache (ip neigh or arp -an) has to keep an entry for every single destination IP address that needs to be contacted, whereas a gateway route only needs one ARP cache entry (for the gateway).

Improve this answer
5
  • Thank you so, so much. I have modified the rule and now use ip route add default via [gateway_ip_here] dev "$interface" table 103 and it works. I should have come for help earlier. At least I learned plenty of things. Cheers Commented Sep 5, 2024 at 19:18
  • I'd recommend doing the same for your other routes as well – even if your gateways can do Proxy-ARP, there are still advantages to a gateway route depending on how much traffic is involved. At the same time, if the USB tethering is there for extended use, I'd get a dedicated LTE modem (like a RUT200 that then connects via Ethernet). Commented Sep 5, 2024 at 19:35
  • Yes, I'm going to do it on the other ones as well to make things cleaner. I suppose this is the reason I see a lot of "who has" messages on my networks, as you were saying ? Another question I would have, although I'll make another post if that's more proper, is the following: using browserleaks.com DNS leak test (from the guest) I see that the DNS that are detected are the one of the ISP to which the host is connected. I guess that's because the host provides the DNS resolution. I'll try adding the gateway IP as DNS server for the connection with nmcli edit. Again thanks a lot, I was lost Commented Sep 5, 2024 at 19:47
  • Yes, omitting the gateway is literally telling the OS that it should make those "who has" ARP queries, just like it does for your actual subnet. The host's Dnsmasq (set up by libvirt) most likely provides DNS resolution – it takes the upstream servers from the host's resolv.conf and advertises itself as the DNS server via DHCP. You can probably do some libvirt network configuration to make it advertise different DNS servers instead of itself. Commented Sep 5, 2024 at 19:55
  • For future readers that might need it: to force the use of a DNS resolver other than the upstream one provided by the host, edit the config of the virtual network. virsh net-edit [virtual_network_name] Then add the following within the <network> config: <dns><forwarder addr="[ip_of_the_gateway]"/></dns> (can't do the proper indentation here, sorry). It solved the DNS leak problem where browserleaks.com detected the ones used by the host. The DNS detected are now indeed those of the ISP of the guest-only NIC (android phone), exactly as I wanted. Thanks again u1686_grawity Commented Sep 5, 2024 at 21:04

You must log in to answer this question.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.