Physical Routing Configuration
Each node has a similar, but slightly different interface and routing configuration. Here is a walkthrough of the
auto lo iface lo inet loopback iface enp5s0f0 inet manual iface enp5s0f1 inet manual iface eno1 inet manual mtu 9000 iface eno2 inet manual mtu 9000 iface eno3 inet manual mtu 9000 iface eno4 inet manual mtu 9000
First, interfaces are defined. Shamrock uses an MTU of 9000 (to allow for jumbo packets) on its physical NICs for internal networks. All interfaces are set to
manual because they are attached to bridges, so that they have the possibility to be bridged to VMs on the host.
auto vmbr0 iface vmbr0 inet static address 192.168.0.20/24 gateway 192.168.0.1 bridge-ports enp5s0f0 bridge-stp off bridge-fd 0
This creates bridge
enp5s0f0 as a bridge port. As described in Physical Configuration.
enp5sf0 is connected to the internet, so
vmbr0 is used as the WAN interface for virtual routers.
auto vmbr1 iface vmbr1 inet static address 10.83.86.10/24 up ip route add 10.83.86.11/32 dev vmbr1 down ip route del 10.83.86.11/32 bridge-ports eno1 bridge-stp off bridge-fd 0 mtu 9000 auto vmbr2 iface vmbr2 inet static address 10.83.86.10/24 up ip route add 10.83.86.12/32 dev vmbr2 down ip route del 10.83.86.12/32 bridge-ports eno2 bridge-stp off bridge-fd 0 mtu 9000
This creates bridges
vmbr2, which respectively have ports
eno2. The reason that there are two seperate bridges for this network is so that per-port static routing works correctly.
Additionally, there are static routes that define routing to the other nodes in the cluster.
vmbr2 are currently unused as VM/LXC bridges.
This static routing configuration is similar to the one described on the Proxmox VE Wiki. There are some differences, such as the use of bridges.
auto vmbr3 iface vmbr3 inet static address 10.83.67.10/24 up ip route add 10.83.67.11/32 dev vmbr3 down ip route del 10.83.67.11/32 up ip route add 10.83.67.131/32 dev vmbr3 down ip route del 10.83.67.131/32 up ip route add 10.83.67.130/32 dev vmbr3 bridge-ports eno3 bridge-stp off bridge-fd 0 mtu 9000 auto vmbr4 iface vmbr4 inet static address 10.83.67.10/24 up ip route add 10.83.67.12/32 dev vmbr4 down ip route del 10.83.67.12/32 up ip route add 10.83.67.132/32 dev vmbr4 down ip route del 10.83.67.132/32 bridge-ports eno4 bridge-stp off bridge-fd 0 mtu 9000
This creates bridges
vmbr4, which respectively have ports
eno4. The configuration is similar to that of the
vmbr2 network, but with additional static routes defined to ensure correct routing to each of the three (3) Kubernetes VMs.
eno4 are connected to the Ceph network,
vmbr4 are added to all three (3) of the Kubernetes nodes.
auto vmbr5 iface vmbr5 inet static address 10.83.87.10/24 bridge-ports none bridge-stp off bridge-fd 0 mtu 9000
The creates bridge
vmbr5, which is specially created so that a proxy LXC container can access services on the Proxmox VE node such as the Proxmox web UI, Ceph Dashboard, and RADOS Gateway. As a consequence it does not have any ports.
vmbr5 allows access to some node services, it is bound to the proxy container of each node.
This line is added to enable the auto-generated interfaces of Software Defined Networking.