Office LAN using two different ISP

So we have in the office a small LAN and two ISPs. Both kind of suck.

ISP1 doesn’t allow SMTP (weird, eh?) and is very slow, not to mention the  frequent disconnects.  ISP2 doesn’t allow VOIP and SSH sessions cannot last more than a few minutes.

As such, we needed a solution where we could route certain types of traffic to the correct ISP.  This post is all about how we did it!

1. First, we needed a Linux machine (Fedora 10) with three network cards. We needed to buy two of them, but they’re very cheap and very straightforward to install. One of them will go to the LAN (eth2) and the other two will go to the WAN (Cablemodem and DSL) routers. Let’s call them eth0 and eth1.
2. We needed to set the kernel to manage routing by having the line:

   net.ipv4.ip_forward = 1

   in the /etc/sysctl.conf file

3. Next, we need to create two independent routing tables, one for each WAN network interface, with their respective default gateway. This default gateway is assigned by the WAN modem/router.  We modify the file /etc/iproute2/rt_tables and add the following lines at the end of the file.

   200 T1

   201 T2

4. Let’s assume that our network configuration is as follows:
   • The LAN is on the 192.168.0.0/16 with a default gateway 192.168.3.1 (also being the address of the Linux machine, because this machine will work as a default gateway for the LAN).
   • WAN1 is on the 10.0.0.0/24 network with 10.0.0.1 as a default gateway and we have the IP address 10.0.0.2 assigned in the eth0 card.
   • WAN2 is on the 172.16.0.0/16 with 172.16.0.1 as a default gateway. The IP assigned to the eth1 card is 172.16.1.33
5. The following script will create the routing tables and NAT configuration so the local network can access the internet.  HTTP, HTTPS, FTP and SMTP will go through WAN1 and SSH, Torrent, and VoIP through WAN2.

   #Default gateway:

   export DEFAULT=eth0

   #Erase the route

   ip route flush cache

   ip route flush table T1

   ip route flush table T2

   ip route del default

   #Erase the firewall rules

   iptables -F

   iptables -X

   iptables -Z

   iptables -t nat -F

   #Interfaces

   export LAN_IF=eth2

   #eth0 = motorola

   export WAN1_IF=eth0

   #eth1 = 2wire

   export WAN2_IF=eth1

   #IP addresses

   export LAN_IP=192.168.3.1

   export WAN_IP1=10.0.0.2

   export WAN_IP2=172.16.1.33

   #Gateways

   export GW1=10.0.0.1

   export GW2=172.16.0.1

   #Networks

   export LAN_NET=192.168.0.0/16

   export WAN1_NET=10.0.0.0/24

   export WAN2_NET=172.16.0.0/16

   if [ $DEFAULT == "eth0" ]

   then

     export DEFAULTGW=$GW1

     export DEFAULTIF=$WAN1_IF

   else

     export DEFAULTGW=$GW2

     export DEFAULTIF=$WAN2_IF

   fi

   #Main routing table

   ip route add $WAN1_NET dev $WAN1_IF src $WAN_IP1 table T1

   ip route add default via $GW1 table T1

   ip route add $WAN2_NET dev $WAN2_IF src $WAN_IP2 table T2

   ip route add default via $GW2 table T2

   #Default gateway

   ip route add default via $DEFAULTGW

   #Or maybe we want to have one packet to each to be sent in round-robin to each interface

   #ip route add default scope global nexthop via $GW1 dev $WAN1_IF weight 1 nexthop via $GW2 dev $WAN2_IF weight 1

   #Routing rules. Choose a routing table

   ip rule add from $WAN_IP1 table T1

   ip rule add from $WAN_IP2 table T2

   #Other recomended rules

   ip route add $LAN_NET     dev $LAN_IF table T1

   ip route add $WAN2_NET     dev $WAN2_IF table T1

   ip route add 127.0.0.0/8 dev lo   table T1

   ip route add $LAN_NET     dev $LAN_IF table T2

   ip route add $WAN1_NET     dev $WAN1_IF table T2

   ip route add 127.0.0.0/8 dev lo   table T2

   #At this point we can see the tables with the command ip route show table T1 (or T2 or main)

   #Firewall marks

   ip rule add fwmark 1 table T1

   ip rule add fwmark 2 table T2

   #Setup default policies to handle unmatched traffic

   iptables -P INPUT ACCEPT

   iptables -P OUTPUT ACCEPT

   iptables -P FORWARD DROP

   #Then we lock our services so they only work from the LAN

   iptables -I INPUT 1 -i ${LAN_IF} -j ACCEPT

   iptables -I INPUT 1 -i lo -j ACCEPT

   iptables -A INPUT -p UDP --dport bootps -i ! ${LAN_IF} -j REJECT

   iptables -A INPUT -p UDP --dport domain -i ! ${LAN_IF} -j REJECT

   #(Optional) Allow access to our ssh server from the WAN1_IF

   iptables -A INPUT -p TCP --dport ssh -i ${WAN1_IF} -j ACCEPT

   iptables -A INPUT -p TCP --dport ssh -i ${WAN2_IF} -j ACCEPT

   #Drop TCP / UDP packets to privileged ports

   iptables -A INPUT -p TCP -i ! ${LAN_IF} -d 0/0 --dport 0:1023 -j DROP

   iptables -A INPUT -p UDP -i ! ${LAN_IF} -d 0/0 --dport 0:1023 -j DROP

   #We add the rules for NAT

   iptables -I FORWARD -i ${LAN_IF} -d ${LAN_NET} -j DROP

   iptables -A FORWARD -i ${LAN_IF} -s ${LAN_NET} -j ACCEPT

   iptables -A FORWARD -i ${WAN1_IF} -d ${LAN_NET} -j ACCEPT

   iptables -A FORWARD -i ${WAN2_IF} -d ${LAN_NET} -j ACCEPT

   iptables -t nat -A POSTROUTING -o ${WAN1_IF} -j MASQUERADE

   iptables -t nat -A POSTROUTING -o ${WAN2_IF} -j MASQUERADE

   #Finally, we make the rules to redirect the traffic to the appropriate interface

   #HTTP and HTTPS via Motorola

   iptables -t mangle -A PREROUTING -p tcp --dport http  -s ${LAN_NET} -j MARK --set-mark 1

   iptables -t mangle -A PREROUTING -p tcp --dport https -s ${LAN_NET} -j MARK --set-mark 1

   #FTP via Motorola

   iptables -t mangle -A PREROUTING -p tcp --dport ftp  -s ${LAN_NET} -j MARK --set-mark 1

   #SMTP via Motorola

   iptables -t mangle -A PREROUTING -p tcp --dport smtp  -s ${LAN_NET} -j MARK --set-mark 1

   #SSH via 2Wire

   iptables -t mangle -A PREROUTING -p tcp --dport ssh  -s ${LAN_NET} -j MARK --set-mark 2

   #Torrent via 2Wire

   iptables -t mangle -A PREROUTING -p tcp --dport 50000:60000  -s ${LAN_NET} -j MARK --set-mark 2

   iptables -t mangle -A PREROUTING -p udp --dport 50000:60000  -s ${LAN_NET} -j MARK --set-mark 2

   #IConnectHere via 2Wire

   iptables -t mangle -A PREROUTING -p tcp --dport 5060:5063  -s ${LAN_NET} -j MARK --set-mark 2

   iptables -t mangle -A PREROUTING -p udp --dport 5060:5063  -s ${LAN_NET} -j MARK --set-mark 2

   iptables -t mangle -A PREROUTING -p tcp --dport 16384:16400  -s ${LAN_NET} -j MARK --set-mark 2

   iptables -t mangle -A PREROUTING -p udp --dport 16384:16400  -s ${LAN_NET} -j MARK --set-mark 2

   iptables -t mangle -A PREROUTING -p tcp --dport 5723  -s ${LAN_NET} -j MARK --set-mark 2

   iptables -t mangle -A PREROUTING -p udp --dport 5723  -s ${LAN_NET} -j MARK --set-mark 2

   iptables -t mangle -A PREROUTING -p tcp --dport 69  -s ${LAN_NET} -j MARK --set-mark 2

   iptables -t mangle -A PREROUTING -p udp --dport 69  -s ${LAN_NET} -j MARK --set-mark 2

6. We also need to configure the DHCP, SAMBA and DNS services so the users of the LAN will get a full experience.

Ta da!!!

1. Ta da!