Under Construction! Please be patient!

Skip to main content

Physical Routing Configuration

Each node has a similar, but slightly different interface and routing configuration. Here is a walkthrough of the /etc/network/interfaces of r710-0.

auto lo
iface lo inet loopback

iface enp5s0f0 inet manual

iface enp5s0f1 inet manual

iface eno1 inet manual
        mtu 9000

iface eno2 inet manual
        mtu 9000

iface eno3 inet manual
        mtu 9000

iface eno4 inet manual
        mtu 9000

First, interfaces  are defined. Shamrock uses an MTU of 9000 (to allow for jumbo packets) on its physical NICs for internal networks. All interfaces are set to manual because they are attached to bridges, so that they have the possibility to be bridged to VMs on the host.

auto vmbr0
iface vmbr0 inet static
        address 192.168.0.20/24
        gateway 192.168.0.1
        bridge-ports enp5s0f0
        bridge-stp off
        bridge-fd 0

This creates bridge vmbr0, with enp5s0f0 as a bridge port. As described in Physical Configuration.

enp5sf0 is connected to the internet, so vmbr0 is used as the WAN interface for virtual routers.

auto vmbr1
iface vmbr1 inet static
        address 10.83.86.10/24
        up ip route add 10.83.86.11/32 dev vmbr1
        down ip route del 10.83.86.11/32
        bridge-ports eno1
        bridge-stp off
        bridge-fd 0
        mtu 9000

auto vmbr2
iface vmbr2 inet static
        address 10.83.86.10/24
        up ip route add 10.83.86.12/32 dev vmbr2
        down ip route del 10.83.86.12/32
        bridge-ports eno2
        bridge-stp off
        bridge-fd 0
        mtu 9000

This creates bridges vmbr1 and vmbr2, which respectively have ports eno1 and eno2. The reason that there are two seperate bridges for this network is so that per-port static routing works correctly.

Additionally, there are static routes that define routing to the other nodes in the cluster.

vmbr1 and vmbr2 are currently unused as VM/LXC bridges.

This static routing configuration is similar to the one described on the Proxmox VE Wiki. There are some differences, such as the use of bridges.

auto vmbr3
iface vmbr3 inet static
        address 10.83.67.10/24
        up ip route add 10.83.67.11/32 dev vmbr3
        down ip route del 10.83.67.11/32
        up ip route add 10.83.67.131/32 dev vmbr3
        down ip route del 10.83.67.131/32
        up ip route add 10.83.67.130/32 dev vmbr3
        bridge-ports eno3
        bridge-stp off
        bridge-fd 0
        mtu 9000

auto vmbr4
iface vmbr4 inet static
        address 10.83.67.10/24
        up ip route add 10.83.67.12/32 dev vmbr4
        down ip route del 10.83.67.12/32
        up ip route add 10.83.67.132/32 dev vmbr4
        down ip route del 10.83.67.132/32
        bridge-ports eno4
        bridge-stp off
        bridge-fd 0
        mtu 9000

This creates bridges vmbr3 and vmbr4, which respectively have ports eno3 and eno4. The configuration is similar to that of the vmbr1 vmbr2 network, but with additional static routes defined to ensure correct routing to each of the three (3) Kubernetes VMs.

 Since eno3 and eno4 are connected to the Ceph network, vmbr3 and vmbr4 are added to all three (3) of the Kubernetes nodes.

auto vmbr5
iface vmbr5 inet static
        address 10.83.87.10/24
        bridge-ports none
        bridge-stp off
        bridge-fd 0
        mtu 9000

The creates bridge vmbr5, which is specially created so that a proxy LXC container can access services on the Proxmox VE node such as the Proxmox web UI, Ceph Dashboard, and RADOS Gateway. As a consequence it does not have any ports.

Since vmbr5 allows access to some node services, it is bound to the proxy container of each node.

source /etc/network/interfaces.d/*

This line is added to enable the auto-generated interfaces of Software Defined Networking.

Back to top