Routing

Routing #

Within this section we will:

  • Examine the fundamentals of “routing”, laying out more clearly what this function is within our network.
  • Install Linux on a machine with several interfaces
  • Perform basic hardening of the machine, as it is on the edge of your network
  • Set up several networks within that machine

Fundamentals #

TODO

  • Fundamentals
    • Forwarding
    • NAT
    • Firewall
  • IPv4/IPv6
  • essential services 
                          ◂            ◂             ◂  (e.g., loopback traffic)
                        ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓ ↙
                        ┃                             ┃
                       ▾┃  INPUT ○╮       ╭○ OUTPUT   ┃▴
                        ┃   hook  │       │  hook    ┏┻┓ ▸      ▸   ▸  outbound
                   ┏━━━━┻━━━━━━━━━┿━━┓ ┏━━┿━━━━━━━━━━┫╳┣━━━━━━┳━━━┿━━━ traffic ▸
                   ┃ ▸    ▸      ▸   ┃ ┃     ▸     ▸ ┗┯┛      ┃▴  │
                   ┃                ▾┃ ┃▴             │       ┃   │
                   ┃▴            local system         │       ┃   ╰○ POSTROUTING
inbound      ▸    ┏┻┓                                 │       ┃      hook
traffic ▸ ━━━━━┿━━┫╳┠── routing decision    routing decision  ┃
               │  ┗┳┛                                         ┃
   PREROUTING ○╯  ▾┃                                          ┃▴
         hook      ┗━━━━━━━━━━━━━┿━━━━━━━━━━━━━━━━━━━━━━━━━━━━┛
                     ▸         ▸ │           ▸           ▸
                                 ╰○ FORWARD hook

Installation #

We will use some specifics from the hardware we’ve selected, however these techniques are generalizable onto whatever platform you have in hand. On the FW6D we have just a single SATA SSD. We will do an installation of Fedora Server using the “netinstall” media, selecting the “minimal” environment group.

  • Software Selection: Minimal
  • Time & Date, Americas/Detroit (or your local time)
  • Root Disabled
  • Create user, ensure user is admin

You will need to start with a display connected to the device, but the goal will be to transition to managing the device via ssh as soon as we can. It’s easier to manipulate the system when working from your workstation environment. We will assume that you are connecting the WAN interface of the device to a network, either your ISP directly or to a staging network where you’re able to pull a routable address.

First Boot #

Login as your user, then become root (sudo su), then we begin!

Set the hostname:

[root@fedora ~]# hostnamectl set-hostname router

Install Necessary packages:

[root@fedora ~]# dnf install screen htop nftables systemd-networkd wireguard-tools vim pciutils usbutils

We will name our interfaces logically, which makes building the firewall later far more intuitive. We will use the MAC address for maximum flexibility. An alternative is to utilize the PCI address, but this can change if the hardware composition is changed (e.g. adding a pci peripheral).

Find the interfaces:

[root@fedora ~]# lspci -D | grep -i ether
0000:01:00.0 Ethernet controller: Intel Corporation I210 Gigabit Network Connection (rev 03)
0000:02:00.0 Ethernet controller: Intel Corporation I210 Gigabit Network Connection (rev 03)
0000:03:00.0 Ethernet controller: Intel Corporation I210 Gigabit Network Connection (rev 03)
0000:04:00.0 Ethernet controller: Intel Corporation I210 Gigabit Network Connection (rev 03)
0000:05:00.0 Ethernet controller: Intel Corporation I210 Gigabit Network Connection (rev 03)
0000:06:00.0 Ethernet controller: Intel Corporation I210 Gigabit Network Connection (rev 03)

The FW6D WAN interface is 0000:01:00.0, then it increments from there where the OPT4 interface is 0000:06:00.0.

We then create the necessary files for systemd to rename the interfaces:

[root@fedora ~]# cd /etc/systemd/network

[root@fedora ~]# vim 0-wan.link 

[Match]
Path=pci-0000:01:00.0

[Link]
Name=wan

[root@fedora ~]# vim 0-lan.link

[Match]
Path=pci-0000:02:00.0

[Link]
Name=lan

We’re going to set up a subnet, and some essential services on the opt1 interface in the event that our bridging is broken it provides us a place to plug into the router directly. We name this interface opt1diag.

[root@fedora ~]# vim 0-opt1.link

[Match]
Path=pci-0000:03:00.0

[Link]
Name=opt1diag

[root@fedora ~]# vim 0-opt2.link

[Match]
Path=pci-0000:04:00.0

[Link]
Name=opt2

[root@fedora ~]# vim 0-opt3.link

[Match]
Path=pci-0000:05:00.0

[Link]
Name=opt3

We’re going to set up another isp via an LTE modem

[root@fedora ~]# vim 0-opt4.link

[Match]
Path=pci-0000:06:00.0

[Link]
Name=opt4lte
TODO
This appears to be different based on distribution, Fedora is consistent where Debian requires manipulation of the initramfs. What is the best way to ensure renames occur consistently?

Now enable and disable services:

[root@fedora ~]# systemctl disable NetworkManager.service
[root@fedora ~]# systemctl disable firewalld.service
[root@fedora ~]# systemctl enable systemd-networkd

We discover that systemd-networkd has a bit of a bug where the unit attempts to start before a dependency and selinux blocks it. We report that here and add a drop in modification:

[root@fedora ~]# mkdir -p /etc/systemd/system/systemd-networkd.service.d/
[root@fedora ~]# vim /etc/systemd/system/systemd-networkd.service.d/after-dbus.conf

[Unit]
After=dbus.socket

We will set up the wan interface so that it will dhcp upon boot. Before this was handled by NetworkManager, but now we’re switching entirely to systemd-networkd.

[root@fedora ~]# vim /etc/systemd/network/wan.network

[Match]
Name=wan

[Network]
DHCP=yes
IPForward=yes

[DHCPv4]
UseDNS=no
UseNTP=no
UseMTU=yes
RouteMetric=100

[DHCPv6]
UseDNS=no
UseNTP=no
RouteMetric=110
  • We want to use DHCP
  • We want to enable kernel packet forwarding (this turns on forwarding for every interface, which is why we must configure a firewall soon)
  • We don’t want the dhcp to push down DNS or NTP to us, we’ll configure that ourselves
  • We set a routing metric on the wan interfaces, allowing us to add more interfaces later (e.g. backup via wireless or satellite broadband)

We now reboot for all these changes to take effect.

Essential Services #

We have the most basic of a system functioning at this point. We should now configure several essential services and harden them as necessary. Upon reboot we should see that the interfaces are all named logically from our .link files we set above:

[root@router ~]# ip -br a
lo               UNKNOWN        127.0.0.1/8 ::1/128 
wan              UP             xx.xx.xx.xx/23 
lan              UP             
opt1diag         DOWN           
opt2             DOWN           
opt3             DOWN           
opt4lte          DOWN

Notice that our wan and lan interfaces are UP as they are connected to devices that have auto-negotiated connection. You should have a routable address on the wan interface. Ideally from here forward you are using that address to manage this device via ssh. If you’re in a pickle to get a routable address on the wan interface skip down to the “Diagnostic Subnet” section below then come back up here to proceed.

Secure Shell #

You should have an ssh keypair on your workstation, if not generate one:

[agd@chonk ~]$ ssh-keygen -t ed25519
  • protect your keypair with a strong passphrase
  • generate a new keypair for each system you login from

Now copy that key to your user on the router:

[agd@chonk ~]$ ssh-copy-id xx.xx.xx.xx

With the key installed, ssh into the router (verify that you are not prompted for a password), and we’ll harden the ssh instance:

[root@router ~]# vim /etc/ssh/sshd_config

Port 4252
Protocol 2
HostKey /etc/ssh/ssh_host_ed25519_key
KexAlgorithms [email protected]
Ciphers [email protected],[email protected],[email protected]
MACs [email protected],[email protected]
AuthenticationMethods publickey
LogLevel VERBOSE
Subsystem sftp  /usr/lib/ssh/sftp-server -f AUTHPRIV -l INFO
PermitRootLogin No
compression No

Before restarting the service we need to inform selinux that we’re going to utilize a non-standard port:

[root@router ~]# dnf install policycoreutils-python-utils

[root@router ~]# semanage port -a -t ssh_port_t -p tcp 4252

We can now restart this service and our changes will be in effect, however the firewall has not been configured. It’s best to get through the next couple sections and restart the entire system.

If seeking more background on hardening ssh the Mozilla Infosec documentation is typically thorough and up to date.

TODO
Consider using oath during login with a TOTP based algorithm.

Domain Name System Resolver #

We will use systemd-resolved as our routers local resolver, as well as a local resolver for each subnet that we construct. We will make some modifications to the existing resolved.conf

[root@router ~]# vim /etc/systemd/resolved.conf

[Resolve]
DNS=1.1.1.1 1.0.0.1 2606:4700:4700::1111 2606:4700:4700::1001
DNSSEC=allow-downgrade
DNSOverTLS=yes
Cache=yes
ReadEtcHosts=yes
DNSStubListener=yes
DNSStubListenerExtra=udp:172.23.23.1:53
DNSStubListenerExtra=udp:172.22.0.1:53
DNSStubListenerExtra=udp:172.22.2.1:53
DNSStubListenerExtra=udp:172.22.4.1:53
DNSStubListenerExtra=udp:172.22.6.1:53
DNSStubListenerExtra=udp:172.22.8.1:53
  • utilize cloudflare as the upstream resolver
  • prefer DNSSEC, but allow for downgrade if necessary
  • Enable DNS over TLS
  • We’d like the router to cache requests, speeds things up
  • We’d like the router to examine its /etc/hosts file for local definitions we place
  • We’d like to use resolved as a listener for the subnets we’re about to construct, so we add an entry for each subnet.

Restart the service for our settings to take effect:

[root@router ~]# systemctl restart systemd-resolved.service

Run a query to see if DNS resolution is functioning:

[root@router ~]# resolvectl query dunn.dev

dunn.dev: 172.67.155.85                        -- link: wan
          104.21.48.178                        -- link: wan
          2606:4700:3031::ac43:9b55            -- link: wan
          2606:4700:3031::6815:30b2            -- link: wan

-- Information acquired via protocol DNS in 145.2ms.
-- Data is authenticated: yes; Data was acquired via local or encrypted transport: yes
-- Data from: network
  • we see that the lookup took 145.2ms
  • we see that the data is authenticated
  • we see that the data was acquired via encrypted transport (DoT)
TODO
If cloudflare supports DNSSEC can’t we just turn it on and not allow for downgrade?

Network Time Protocol #

Systemd ships with systemd-timesyncd but it will only function as an NTP client. We will provide NTP services from the router by using Chrony which functions as an NTP client and server.

We will make some small modifications to the existing chrony configuration:

[root@router ~]# vim /etc/chrony.conf

pool time.cloudflare.com iburst nts
allow 172.22.0.0/16
  • utilize cloudflare time servers, encrypted with Network Time Security
  • allow the 172.22.0.0/16 subnet to access the chrony server instance

Ensure that the timezone is set properly, even though we did set it during the install:

[root@router ~]# timedatectl set-timezone America/Detroit

Restart the chrony daemon:

[root@router ~]# systemctl restart chronyd.service

Examine chrony status:

[root@router ~]# chronyc sources

MS Name/IP address         Stratum Poll Reach LastRx Last sample               
===============================================================================
^+ time.cloudflare.com           3  10   377   719   +560us[ +560us] +/-   20ms
^* time.cloudflare.com           3  10   377   814   +553us[ +605us] +/-   20ms

Verify that NTS is in effect:

[root@router ~]# chronyc authdata

Name/IP address             Mode KeyID Type KLen Last Atmp  NAK Cook CLen
=========================================================================
time.cloudflare.com          NTS     1   15  256  10d    0    0    8  100
time.cloudflare.com          NTS     1   15  256  10d    0    0    8  100
TODO
Provide NTP via TLS with a legitimate certificate.

Building Networks #

We build out the interfaces that we’d planned earlier. We will make heavy use of the systemd.network and systemd.netdev documentation.

We will use systemd-networkd internal implementation of a DHCP server which combined with systemd-resolved DNSStubListener allows us to handle DHCP and DNS requests without having another daemon like dnsmasq. There are two current downsides:

  • systemd-networkd doesn’t allow for static lease, yet, there is a merged PR here which will arrive in the next release.
  • systemd-resolved doesn’t allow for a domain based splitting of resolve servers like dnsmasq, documented here.

systemd-networkd configuration files can seem bewildering at first, but after inspection they eventually become intuitive. With the diagnostic subnet we will be associating a configuration directly with an interface. With the other subnets we will be creating VLAN(s) and associating those to the lan interface. As we have a plan, we can write out the association of VLAN(s) to lan right now:

[root@router ~]# vim /etc/systemd/network/lan.network

[Match]
Name=lan

[Network]
DHCP=no
LinkLocalAddressing=no
VLAN=management
VLAN=home
VLAN=guests
VLAN=things
VLAN=work

We don’t want the lan interface to have an address. It’s just an interface that we’re going to hang VLAN(s) on. Now let’s build out all the networks and necessary VLAN interfaces.

Diagnostic Subnet #

This subnet will “hang” from the physical opt1 interface on the FW6D, we renamed it to opt1diag earlier.

[root@router ~]# vim /etc/systemd/network/diagnostic.network

[Match]
Name=opt1diag

[Network]
Address=172.23.23.1/24
DHCPServer=yes

[DHCPServer]
PoolOffset=20
DNS=172.23.23.1
NTP=172.23.23.1
Timezone=America/Detroit
EmitDNS=yes
EmitNTP=yes
EmitRouter=yes
EmitTimezone=yes

Management Subnet #

We create the vlan interface:

[root@router ~]# vim /etc/systemd/network/management.netdev

[NetDev]
Name=management
Kind=vlan

[VLAN]
Id=220

We associate a network with that interface:

[root@router ~]# vim /etc/systemd/network/management.network

[Match]
Name=management

[Network]
Address=172.22.0.1/24
DHCPServer=yes

[DHCPServer]
PoolOffset=100
DNS=172.22.0.1
NTP=172.22.0.1
Timezone=America/Detroit
EmitDNS=yes
EmitNTP=yes
EmitRouter=yes
EmitTimezone=yes

Home Subnet #

We create the vlan interface:

[root@router ~]# vim /etc/systemd/network/home.netdev

[NetDev]
Name=home
Kind=vlan

[VLAN]
Id=222

We associate a network with that interface:

[root@router ~]# vim /etc/systemd/network/home.network

[Match]
Name=home

[Network]
Address=172.22.2.1/24
DHCPServer=yes

[DHCPServer]
PoolOffset=100
DNS=172.22.2.1
NTP=172.22.2.1
Timezone=America/Detroit
EmitDNS=yes
EmitNTP=yes
EmitRouter=yes
EmitTimezone=yes

Guests Subnet #

We create the vlan interface:

[root@router ~]# vim /etc/systemd/network/guests.netdev

[NetDev]
Name=guests
Kind=vlan

[VLAN]
Id=224

We associate a network with that interface:

[root@router ~]# vim /etc/systemd/network/guests.network

[Match]
Name=guests

[Network]
Address=172.22.4.1/24
DHCPServer=yes

[DHCPServer]
PoolOffset=100
DNS=172.22.4.1
NTP=172.22.4.1
Timezone=America/Detroit
EmitDNS=yes
EmitNTP=yes
EmitRouter=yes
EmitTimezone=yes

Things Subnet #

We create the vlan interface:

[root@router ~]# vim /etc/systemd/network/things.netdev

[NetDev]
Name=things
Kind=vlan

[VLAN]
Id=226

We associate a network with that interface:

[root@router ~]# vim /etc/systemd/network/things.network

[Match]
Name=things

[Network]
LinkLocalAddressing=no
Address=172.22.6.1/24
DHCPServer=yes

[DHCPServer]
PoolOffset=100
DNS=172.22.6.1
NTP=172.22.6.1
Timezone=America/Detroit
EmitDNS=yes
EmitNTP=yes
EmitRouter=yes
EmitTimezone=yes

Work Subnet #

We create the vlan interface:

[root@router ~]# vim /etc/systemd/network/work.netdev

[NetDev]
Name=dod
Kind=vlan

[VLAN]
Id=2222

We associate a network with that interface:

[root@router ~]# vim /etc/systemd/network/work.network

[Match]
Name=dod

[Network]
LinkLocalAddressing=no
Address=172.22.22.1/24
DHCPServer=yes

[DHCPServer]
PoolOffset=100
DNS=208.67.222.222
NTP=172.22.22.1
Timezone=America/Detroit
EmitDNS=yes
EmitNTP=yes
EmitRouter=yes
EmitTimezone=yes

Notice that we’re setting the DNS to OpenDNS because our “work” does a terrible job with DNS and for whatever reason OpenDNS is pretty permissive of their badness.

Firewall #

Now the final piece to make the router fully functional.

[root@router ~]# vim /etc/sysconfig/nftables.conf

flush ruleset

table ip nat {
   chain prerouting {
      type nat hook prerouting priority 0;
   }
   chain postrouting {
      type nat hook postrouting priority 100;

      oifname wan masquerade
   }
}

table inet filter {

    chain base_checks {
        # allow established/related connections
        ct state {established, related} accept

        # early drop of invalid connections
        ct state invalid drop
    }

   set meter_ssh {
      type ipv4_addr
      size 65535
      flags dynamic
    }  

   chain input {
      type filter hook input priority 0; policy drop;
      
      jump base_checks
      
      iifname { lo } accept

      # allow icmp and igmp
      ip6 nexthdr icmpv6 icmpv6 type { echo-request, echo-reply, packet-too-big, time-exceeded, parameter-problem, destination-unreachable, packet-too-big, mld-listener-query, mld-listener-report, mld-listener-reduction, nd-router-solicit, nd-router-advert, nd-neighbor-solicit, nd-neighbor-advert, ind-neighbor-solicit, ind-neighbor-advert, mld2-listener-report } accept
      ip protocol icmp icmp type { echo-request, echo-reply, destination-unreachable, router-solicitation, router-advertisement, time-exceeded, parameter-problem } accept    
      ip protocol igmp accept
	
      # dns (53), dhcp (67,68), ntp (123)
      iifname { opt1diag, management, home, guests, things, work } ip protocol udp udp dport { 53, 67, 68, 123 } counter accept

      # ssh from trusted
      iifname { opt1diag, home } ip protocol tcp tcp dport 4252 counter accept

      # rate limit bare ssh from wan
      iifname wan ip protocol tcp tcp dport 4252 ct state new add @meter_ssh { ip saddr limit rate 5/minute burst 3 packets } counter accept
   }

   chain forward {
      type filter hook forward priority 0; policy drop;
      
      jump base_checks

      # Path MTU Discovery MSS Clamping
      tcp flags syn tcp option maxseg size set rt mtu
      
      # wan bound
      iifname opt1diag oifname wan accept
      iifname home oifname wan accept
      iifname management oifname wan accept
      iifname guests oifname wan accept
      iifname work oifname wan accept

      # inter vlan
      iifname home oifname management accept
      iifname home oifname guests accept
      iifname home oifname things accept
   
    }
    chain output {
      type filter hook output priority 0; policy accept;
    }
}

Let’s unpack, first open this up, then:

  • iifname: “Input Interface Name”
  • oifname: “Output Interface Name”
  • table ip nat
  • table inet filter
    • set up base_checks, which we use a jump to get to.

      If you use jump to get packet processed in another chain, packet will return to the chain of the calling rule after the end

    • set up meter_ssh, which we use to rate limit a little later.

      Dynamic sets/maps or meters are a way to use maps with stateful objects.

    • chain input
      • jump base_checks
      • allow everything on the loopback interface
      • allow ICMP and IGMP
      • allow DNS, DHCP, and NTP coming from interfaces opt1diag, management, home, guests, things, work
      • allow ssh, without rate limiting, from interfaces opt1diag, home
      • allow ssh from wan, but rate limit it using meter_ssh
    • chain forward
      • jump base_checks
      • tcp flags syn tcp option maxseg size set rt mtu Path MTU Discovery MSS Clamping
      • allow interface opt1diag, home, management, guests, work to forward to the wan interface
      • allow interface home to initiate connections to interface management
      • allow interface home to initiate connections to interface guests
      • allow interface home to initiate connections to interface things
    • chain output

Now if you reboot you have your first take on an edge-of-the-network routing and firewalling platform.

Port Forwarding and NAT Loopback/Hairpin #

Building on the above example let us go a bit further, we will set up Port Forwarding and NAT Loopback.

Port forwarding is an application of NAT that redirects a communication request from one address and port number to another while the packets are traversing a router. This is used to make services on a host inside a masqueraded network exposed on the outside of the router.

NAT Loopback is a technique that allows for the utilization of a service via the public address of the overall masquerad’ed network(s). The alternative is to use DNS to provide localized resolution to the service when inside the network, however this is becoming more difficult with DoT and DoH. There is a comprehensive write up here, and theory explained here.

In our example we’ll say that we have:

  • a local client at 172.22.2.63, somewhere in the 172.22.2.0/24 subnet
  • a webserver listening on port 80 and 443 at 172.22.2.10
  • a public “wan” address of 66.66.66.66

Our goal will be to facilitate the local client and a remote client to access this webserver by using NAT loopback rather than separate DNS entries for the local and non-local network. We will modify the firewall from the example above:

[root@router ~]# vim /etc/sysconfig/nftables.conf


define wan = 66.66.66.66
define server = 172.22.2.10

table ip nat {
   chain prerouting {
      type nat hook prerouting priority 0;

      # dnat for tcp http, https to our server
	   ip daddr $wan tcp dport { 80, 443 } dnat to $server

      # hairpin for http, https to server
	   ip saddr 172.22.2.0/24 ip daddr $wan tcp dport { 80, 443 } dnat to $server
   }
   chain postrouting {
      type nat hook postrouting priority 100;

      oifname wan masquerade

      # hairpin response from server through router
      ip saddr 172.22.2.0/24 ip daddr $server tcp dport { 80, 443 } snat to 172.22.2.1
   }
}
table inet filter {
   chain forward {
      type filter hook forward priority 0; policy drop;

      # forwarded server
	   ip daddr $server ct status dnat accept
    }
}
  • table ip nat
    • chain prerouting
      • ip daddr $wan tcp dport { 80, 443 } dnat to $server: translating packets coming to our router for (80,443) to our server (half of port forwarding)
      • ip saddr 172.22.2.0/24 ip daddr $wan tcp dport { 80, 443 } dnat to $server: translating packets coming from our home subnet, to our router for (80,443) to our server (half of NAT loopback)
    • chain postrouting
      • ip saddr 172.22.2.0/24 ip daddr $server tcp dport { 80, 443 } snat to 172.22.2.1: translating packets coming from our home subnet, to our server, with a source NAT that informs our server that when it replies to the client from our home subnet to send it through the home subnet router (other half of NAT loopback)
  • table inet filter
    • chain forward
      • ip daddr $server ct status dnat accept: accept destination NAT for our server (other half of port forwarding)