pebkac routing problem

[Martin P. Hellwig]

Martin P. Hellwig wrote:
<cut problem>
Thanks to hints from Matt I solved my routing problem concerning 
multiple gateways on non routing uplinks using IPFW, I scribbled my 
progress down and like to share that with you guys.

Here you go:

Description:
The limitation of a single default gateway in combination with a multi 
homed computer on non routing networks will prevent the other connection 
from working correctly.

The problem is that when a IP traffic is initiated from an outside 
source via the other route, the reply is still done over the default 
gateway.
All correctly configured gateways on a non routing network will filter 
these packages out because the gateway has no knowledge that these 
originating IP addresses are not spoofed.

The solution is to use the gateway related to the IP address/network 
where your packages is coming from.
There are different ways to achieve this most of them rely on a package 
filter/manager, others on multiple routing tables.

On my DragonFlyBSD system IPFW is suitable to do this task, for FreeBSD 
it should be the same.

First I need to configure my system to use IPFW:

% echo 'firewall_enable="YES"'>> /etc/rc.conf
% echo 'firewall_script="/etc/rc.firewall"' >> /etc/rc.conf
% echo 'firewall_type="OPEN"' >> /etc/rc.conf
The last line prevents me from logging myself out permantly and lets the 
machine behave as if there is no firewall. In /etc/rc.firewall "Open" is 
define with a couple of (indirect) rule sets.
Although the machine is within immidialty physical reach (as it should 
if you're doing network adaption) I administrate it over a secure remote 
shell.

If you have a machine doing NAT you probably have a recompiled kernel 
with built in IPFW and "options IPDIVERT", thus IPFW is already present 
on your system and you don't need to configure the above part. If you 
have other firewall/nat configuration, this explanation might do you 
more harm then good.

starting IPFW if it's not already started:
% /etc/rc.d/ipfw start
At this point all network connections are dropped and I have to relogon.
Let's see what the current "Open" IPFW rules are:
% ipfw list
00100 allow ip from any to any via lo0
00200 deny ip from any to 127.0.0.0/8
00300 deny ip from 127.0.0.0/8 to any
65000 allow ip from any to any
65535 deny ip from any to any
Now I have to add a rule that when my 'other' IP address sends a package 
back that it does that over the 'other' gateway except when it's a 
address on the local bus. But first I need to configure my second IP 
address and get some info from my existing configuration.

My configuration for my default (and first) network:
IP and netmask 145.103.249.189/27 with gateway 145.103.249.190
The other network I like to add:
213.126.48.226/24 with gateway 213.126.48.1
Both networks are directly reachable from one network card (fxp0) i.e. 
the same physical network, so I added a alias:
% ifconfig fxp0 alias 213.126.48.226/24
At this moment all machines can reach my machine over the added IP 
address and only machines on 213.126.48.0/24 receive my reply 
originating from the added alias.
If you have your second network on a separate physical network, 
configure your NIC as usual.

With this information the rules I created and executed are:
% ipfw add 110 fwd 213.126.48.1 ip from 213.126.48.226 to not 
213.126.48.0/24
% ipfw add 120 fwd 145.103.249.190 ip from 145.103.249.189 to not 
145.103.249.160/27

Strictly speaking the last rule is not (yet) necessary because thats the 
behavior of the default gateway however in my case, my default gateway 
will change so I included the rule already now.
Do notice that this is an unnecessary system burden although probably 
not too much!

At this moment my machine does exactly what I want, however if I reboot 
it, all configurations are lost, to make it permantly:
echo 'ifconfig_fxp0_alias0="inet 213.126.48.226/24"' >> /etc/rc.conf

Then open /etc/rc.conf in a editor and searched for the definition of 
"Open".
I found it at:
case ${firewall_type} in
    [Oo][Pp][Ee][Nn])
        allow_loopback
        deny_spoof
        divert_nat
        allow_rest
    ;;

I added the below rules under "allow_rest":
${fwcmd} add 110 fwd 213.126.48.1 ip from 213.126.48.226 to not 
213.126.48.0/24
${fwcmd} add 120 fwd 145.103.249.190 ip from 145.103.249.189 to not 
145.103.249.160/27

So that the configuration is now:
case ${firewall_type} in
    [Oo][Pp][Ee][Nn])
        allow_loopback
        deny_spoof
        divert_nat
        allow_rest
        ${fwcmd} add 110 fwd 213.126.48.1 ip from 213.126.48.226 to not 
213.126.48.0/24
        ${fwcmd} add 120 fwd 145.103.249.190 ip from 145.103.249.189 to 
not 145.103.249.160/27
    ;;

--
mph