carp configuration on openbsd/pf

Typically, when a service provider gives you your IP's (say, for a T1 service) they give you both a WAN and LAN range. This can be confusing, because typically a LAN-range would imply private (RFC1918) IP addressing, however telcos dont think of it that way. The WAN range is typically a /30, which is appropriate for a point-to-point topology. Its implied that this IP be used on the CPE, which has been traditionally a router. The LAN range is typically a /27, and these IP's could be used for services that may be exposed through the router/firewall, but it could be done the same with the WAN IP and use port redirection to anything we place in the dmz in the future. This latter design approach is the one I chose. The less IP's the smaller the attack surface and therefore less to manage. The important part to note about this configuration is that the CARP IP address and the real ip of the physical NIC do not need to be in the same subnet or related in any way. Another thought I had was that instead of using the LAN IP's as the real IP's on the public facing interfaces of the firewall, you could instead use RFC1918 addressing and it would ensure that any mistake (unlikely) in routing traffic outside of the interface would be dropped at the next hop.

The specific service I contracted was through XO, its a 10Mb pipe to the Internet with an Ethernet hand-off. They gave me a Hatteras 408-CPi. I decided that we didn't need an extra hop on our side, so I eliminated the router and plugged the XO service straight into my firewall (basically) The blue cable coming into the switch is from the Hatteras unit. The yellow cables goto the outside nic's on each firewall. The grey goto the firewall's dmz nic's. The single yellow cable on the second switch goes to a ethernet port on a router that figures out what network the traffic is truly destined for. The orange, as you probably guessed, is a crossover cable, all pfsync data is transmitted on this interface.

In this implementation I used two DL360's, one more powerful than the other, both with 2GB ram. I also used two Cisco 2960G-24TS switches.

Configuration steps for both hosts for CARP with pf

Edit /etc/sysctl.conf

Permit forward routing of IP packets
net.inet.ip.forwarding=1

Allow CARP to function
net.inet.carp.allow=1
net.inet.carp.log=1

If one interface fails, then failover to second host
net.inet.carp.preempt=1

Kernel will not respond to incoming connections to unbound network ports
net.inet.tcp.blackhole=2 # Drop incoming packets to unbound tcp ports
net.inet.udp.blackhole=1 # Drop incoming packets to unbound udp ports

Force a reboot if the kernel panics
ddb.panic=0


Individual Host Configuration for CARP with pf

REXFW01 - Primary
-------------------------
bge0: [private_ip]
bge1: 222.222.222.221 + multicast
bge2: 10.10.2.2

#> ifconfig pfsync0 syncdev bge1
#> ifconfig pfsync0 up
#> ifconfig carp0 create
#> ifconfig carp0 vhid 1 pass xxx carpdev bge0 [carp_ip] netmask 255.255.255.252
#> ifconfig carp1 create
#> ifconfig carp1 vhid 2 pass xxx carpdev bge2 10.10.2.1 netmask 255.255.255.0


Create the permanent config files

#> create /etc/hostname.pfsync0 up syncdev bge1
#> create /etc/hostname.carp0 inet [carp_ip] 255.255.255.252 [gateway] vhid 1 pass xxx carpdev bge0
#> create /etc/hostname.carp1 inet 10.10.2.1 255.255.255.0 10.10.2.255 vhid 2 pass xxx carpdev bge2


REXFW02 - Secondary
-------------------------
bge0: [private_ip]
bge1: 222.222.222.222 + multicast
bge2: 10.10.2.3

#> ifconfig pfsync0 syncdev pcn1
#> ifconfig pfsync0 up
#> ifconfig carp0 create
#> ifconfig carp0 vhid 1 pass xxx advskew 100 carpdev bge0 [carp_ip] netmask 255.255.255.252
#> ifconfig carp1 create
#> ifconfig carp1 vhid 2 pass xxx advskew 100 carpdev bge2 10.10.2.1 netmask 255.255.255.0


Create the permanent config files

#> create /etc/hostname.pfsync0 up syncdev bge1
#> create /etc/hostname.carp0 inet [carp_ip] 255.255.255.252 [gateway] vhid 1 advskew 100 pass xxx carpdev bge0
#> create /etc/hostname.carp1 inet 10.10.2.1 255.255.255.0 10.10.2.255 vhid 2 advskew 100 pass xxx carpdev bge2

-------------------------
pf.conf - for both primary and secondard firewalls
-------------------------
# $OpenBSD: pf.conf,v 1.34 2007/02/24 19:30:59 millert Exp $
#
# See pf.conf(5) and /usr/share/pf for syntax and examples.
# Remember to set net.inet.ip.forwarding=1 and/or net.inet6.ip6.forwarding=1
# in /etc/sysctl.conf if packets are to be forwarded between interfaces.

#Interface Macros
loopback="lo0"
ext_if="bge0"
int_if="bge2"
pfsync_if="bge1"
ext_carp="carp0"
int_carp="carp1"

# Allowed incoming ICMP types
icmp_types = "{ echorep, echoreq, timex, paramprob, unreach code needfrag }"

# Private networks (RFC 1918)
priv_nets = "{ 127.0.0.0/8, 192.168.0.0/16, 172.16.0.0/12, 10.0.0.0/8 }"

# Internal Networks
our_nets = "{ 10.10.2.0/24, 10.10.1.0/24 }"

#DNS Servers
dns_srv = "{ x.x.x.x, x.x.x.x }"

#table <spamd-white> persist
#set skip on lo

scrub in all

#NAT outgoing traffic
nat on $ext_if from $our_nets to any -> ($ext_carp)

#nat on $ext_if from $int_if:network to any -> ($ext_carp)
rdr pass log on $ext_if inet proto tcp from any to $ext_carp port 44 -> $int_if port 22

# Block all incoming traffic from external interface
block log all
block drop in on $ext_if all
block drop in quick from urpf-failed

# Block incoming traffic from private networks on external interface
block drop in quick on $ext_if from $priv_nets to any

# Block outgoing traffic to private networks on external interface
block drop out quick on $ext_if from any to $priv_nets

#Allow CARP and pfsync protocols
pass quick on $pfsync_if proto pfsync
pass quick proto carp

pass quick on $loopback inet proto tcp from $loopback to $loopback port smtp keep state

#Allow outgoing traffic
pass in on $int_if from $our_nets to any
pass out on $int_if from any to $our_nets
pass out on $ext_if proto tcp all modulate state flags S/SA
pass out on $ext_if proto { icmp, udp } all keep state

# Allow ICMP echos
pass in on $int_if inet proto icmp icmp-type 8 code 0 keep state
pass out on $int_if inet proto icmp icmp-type 8 code 0 keep state

#Allow DNS lookups
pass in on $ext_if proto udp from $our_nets to any port 53 keep state
pass out on $ext_if proto udp from $our_nets to any port 53 keep state

#Allow SNMP
pass quick on $int_if proto udp from $our_nets to $int_if port 161 keep state
antispoof log quick for $int_if
antispoof log quick for lo