Under Construction! Please be patient!

Skip to main content

Physical Configuration

Each of the Dell PowerEdge r710 servers in Shamrock is configured with 6 NICs:

  • 4 of these are internal Broadcom NetXtreme II BCM5709 Gigabit Ethernet NICs (eno1, eno2, eno3, eno4)
  • 2 of these are from a PCI-E add-in card:
    • r710-0 uses a Intel Corporation 82576 Gigabit NIC (2 ports)
    • r710-1 and r710-2 use Broadcom NetXtreme II BCM5709 Gigabit Ethernet NICs (2 ports)

There is now an additional Broadcom NetXtreme II BCM5709 PCI-E add-in card in inventory, and it is planned to be installed into r710-0 in the future to replace the Intel card.

Additionally, each r710 has an integrated iDRAC enterprise NIC for the baseboard management controller. Because iDRAC 6 does not support modern encryption standards, it is necessary to take care that its traffic does not directly come into contact with the other Shamrock networks.

Currently, iDRAC NICs are wired directly into the server, staggered from one server to another, so that even if a node is powered down, another node will still be able to access that node's iDRAC.

There are plans to replace these staggered connections in the future with an additional network switch or VLAN to better support this high availability.

Full Mesh Networking

Full mesh networking is generally only useful in cases where there are few nodes inside of the mesh, and network switches are not available (i.e. 10 gigabit networks between three (3) HA nodes). This is because the number of NICs and connections between nodes required dramatically increases the more nodes you add. The usage of 1 gigabit full mesh networking on Shamrock is born from a lack of gigabit switches on hand and as an exercise in networking.

Shamrock employs the use of direct node-to-node connections and static routing to create two (2) physical networks, shown in blue and purple in the diagram above. The first network (blue, eno1, eno2) is used for Proxmox cluster communication, and as the Software Defined Networking transport layer. The second network (purple, eno3, eno4) is used exclusively as the Ceph network.

More information on network configuration for full mesh networking can be found in Physical Routing Configuration.

Back to top