How to set up two NICs in same machine different subnets and route from one to the other?

Background

A netmask (or just "mask" for short) defines how to split an IP address into a network part and a host part. The IP protocol assumes by default that a network interface will be able to directly contact a remote host if the remote host's IP address and the interface's IP address both have a matching network part. If the network part is different, then the system needs to check its routing table to figure out what to do.

An IPv4 netmask is fully expressed as a 32-bit value which has all 1-bits starting from the most significant bit until some bit position, then all 0-bits. The 1-bits indicate the corresponding bit of the IP address is part of the network part of the address; 0-bits indicate the host part, respectively. As a result, it is convenient to describe a netmask by just the number of 1-bits it has; this can be properly known as netmask length, but often is referred just as "mask" for brevity. So, a mask of /24 is fully written out as 255.255.255.0 in the normal base-10 IP address format, or 11111111.11111111.11111111.00000000 in binary.

If the netmask length is divisible by 8, it makes the separation of the network part and host part align with the base-10 representation of the IP address, which makes it easy for humans: for example, 192.168.0.251/24 means an IP address where the network part is 192.168.0 and the host part is .251.

Netmasks that are not divisible by 8 require a tiny bit of binary math to understand. But in a nutshell, 192.168.0.0/23 indicates a network whose first address is 192.168.0.0 and the last address is 192.168.1.255, so one of your networks would be a subnetwork of the other. On the other hand, your ip addr indicates both eth0 and eth1 have a mask of /24 configured. The routing table entries with no gateway (in route output) or via x.x.x.x (in ip r) should match the configuration of the network interfaces, so there must be a typo or some other error somewhere. The ip addr output you copied is not consistent with the ip r/route outputs.

---

A routing table is processed starting from most specific entries (= those with the largest netmask) towards the more generic ones (smaller netmask values). A default gateway entry is the most generic one, matching every address that has not been matched by any other entries. Technically it can be expressed as an all-zeroes IP address with a mask length of /0.

In human terms, a default gateway entry in the routing table is "if you can't directly reach some address and there isn't any more specific instructions, send it to this directly-reachable system; it will know what to do with it."

When you configure an IP address and netmask to a network interface, the Linux kernel will automatically generate a routing table entry that matches the hosts directly reachable by that interface, according to the netmask. So if your system has two network interfaces plugged to different network segments, and no Internet connection, then that system needs no manual route entries at all, and no default gateway either. But the other hosts in those two networks will need a routing table entry to know that the "other" network can be reached by routing through the 2-interface host.

If there's just the locally-reachable network and the other network, a default gateway entry pointing at the 2-interface host would suffice; but if there is also an Internet connection involved, the default gateway needs to point at your Internet router (because all the other network segments of the world are reachable through it and you most definitely don't want to enumerate them one by one) and if the resulting default gateway entry does not point to your 2-interface host, you'll also need a specific route entry that tells the system "my other network segment has this network address and this netmask, and can be reached through the 2-interface host".

---

What you'll need to do

Assuming that your ip addr output is the correct one and you have two distinct network segments, one 192.168.0.x/24 and another 192.168.1.x/24, you need to do four things to enable routing between them:

1.) Enable IPv4 routing master switch on the CM4.

echo "net.ipv4.ip_forward=1" >>/etc/sysctl.conf sysctl -p 

2.) Ensure that the firewall rules on the CM4 allow forwarding, i.e. iptables -L FORWARD -vn either has policy ACCEPT and no rules, or suitable ACCEPT rules in place if the policy is DROP. Some distributions set the forwarding policy to DROP by default. If that's the case, you'll need two rules:

iptables -I FORWARD 1 -s 192.168.0.0/24 -d 192.168.1.0/24 -j ACCEPT iptables -I FORWARD 2 -s 192.168.1.0/24 -d 192.168.0.0/24 -j ACCEPT 

In human terms: packets incoming from 192.168.0.x with a destination in 192.168.1.x will be accepted for forwarding (routing), and vice versa.

To make these rules persistent, you could e.g. run iptables-save and redirect the output to a file appropriate for your distribution. If your system is using ufw or some other firewall configuration tool, use that instead to configure equivalent persistent rules.

3.) On the hosts in the 192.168.0.x private LAN: set their default gateway to 192.168.0.60. Since there is no Internet gateway in that segment, that's the only route entry they'll need. (If you later want to enable outgoing Internet access from the private LAN, you will be able to do it by just changing the iptables FORWARD rules on the CM4.)

4.) On the hosts in the 192.168.1.x main network: these should already have a default gateway setting of 192.168.1.1, and if you change it, these hosts will lose their Internet connectivity. So here, you'll need to define a specific route that says "the network 192.168.0.x with mask /24 will be reachable by routing through 192.168.1.200".

With Linux configuration commands, this is either:

ip route add 192.168.0.0/24 via 192.168.1.200 

or

route add -net 192.168.0.0 netmask 255.255.255.0 gw 192.168.1.200 

Other operating systems may need a slightly different syntax.