The fundamental problem here is that when your system is forwarding traffic (that is, acting as a router for other nodes on your network), it has no idea which address the other node was using as the default gateway. Whether you have set 192.168.0.1 or 192.168.0.5 as the default gateway on other nodes, the process goes something like this:
- Node tries to connect to a remote address (say, 8.8.8.8)
- Node realizes it has no direct route to 8.8.8.8, so it checks for a default route
- Hooray, there is a default route!
- Node sends an ARP request for the default gateway address if it doesn't already have the MAC address in its cache
- Node sends a packet to 8.8.8.8 with the MAC address of the gateway as the next hop
This means that when your router receives that outbound packet, nothing in the packet identifies which address the node used to determine the MAC address of the gateway.
So, what can you do?
The easiest solution is to ensure that all the nodes on your "restricted" network have addresses allocated from an address range that is different from that of your "unrestricted" network. For example, if your network is 192.168.0.0/24, then maybe you allocated unrestricted addresses from 192.168.0.0/25 and restricted addresses from 192.168.0.128/25. This will require some work with your DHCP server -- a common configuration would be to create static entries for the known hosts on your restricted network, and then arrange for unknown hosts to receive addresses from the restricted range.
With this configuration in place, you can use the origin address of connections in your forwarding rules:
iptables -A FORWARD -s 192.168.0.128/25 -j DROP -m comment --comment "Drop connections from restricted hosts" iptables -A FORWARD -s 192.168.0.0/25 -j ACCEPT -m comment --comment "Forward connections from unrestricted hosts"
If you have the network equipment to support it, you could move the restricted network to a separate VLAN; this would allow you to have a completely separate DHCP server for restricted hosts and would allow you to use source network or source interface in your filtering rules.