Home > Tech Gallery > Technology Feast | Distributed VXLAN Implementation Solution Based on EVPN
Time: September 19th, 2024
This article will describe the currently popular EVPN+VXLAN distributed implementation solution along with the development process of VXLAN.
Why adopt a distributed VXLAN solution?
Currently, there are two main ways to implement VXLAN: centralized and distributed. So should we choose a centralized VXLAN solution or a distributed VXLAN solution in the VXLAN deployment solution? In fact, this question is easy to answer. You can choose according to the scale of the data center. As shown in the following table, the best practice for large data centers with more than 5,000 server clusters is the distributed VXLAN solution.
|
Centralized VXLAN
|
Distributed VXLAN
|
|
|
Layer 3 gateway location
|
Layer 3 gateway is deployed at Spine
|
Layer 3 gateway is deployed at Leaf
|
|
Layer 2 gateway location
|
Layer 2 gateway is deployed at Leaf
|
Layer 2 gateway is deployed at Leaf
|
|
Traffic forwarding rules
|
Layer 3 traffic under the same -Leaf needs to detour the Spine
|
Layer 3 traffic under the same -Leaf does not need to detour the Spine
|
|
Table item storage rule
|
The Spine needs to store the MAC and ARP of the entire network
|
The Spine does not need to store the MAC and ARP of the entire network
|
|
Control Plane
|
Static or EVPN
|
EVPN
|
|
Operation and Management (O&M)
|
Simple, because the gateway is at Spine
|
Complex, because the gateway is at Leaf
|
|
Applicable Data center scale
|
Suitable for small scale networking
|
Suitable for medium and large scale networking
|
▲ Table 1 Differences between centralized VXLAN and distributed VXLAN
The centralized approach will have the common problems of traditional large Layer 2 networks, such as broadcast storms. To avoid this problem, the broadcast domain should not be too large, and thus the cluster size should not be too large as well. However, the distributed approach can reduce the broadcast domain to the edge because of Layer 3 along with the control function of EVPN, thereby forming a larger network scale.
Why EVPN is introduce as the control plane?
By comparison, distributed gateways are more suitable for networking with public cloud vendors or medium-to-large-scale private cloud data centers. VXLAN is defined by RFC7348, which only defines the behavior of the forwarding plane and does not specify the VXLAN control plane. In the early days of VXLAN technology, forwarding information was obtained through the forwarding plane, which was relatively simple to implement. However, as the scale of the network grows, relying entirely on the forwarding plane for control will cause broadcast and multicast storms in the network, and bring great problems to operation and maintenance (O&M). Therefore, VXLAN also needs a control plane. Before understanding EVPN, let's take a look at the comparison of several VXLAN control planes, as shown in the following table:
|
Static mode
|
IS-IS mode
|
EVPN mode
|
Controller mode
|
|
|
Tunnel Construction method
|
Adopts a static configuration approach
|
ENDP(Enhanced Neighbor Discovery Protocol) must be configured on VTEPs to discover VTEPs and establish tunnels
|
The extended BGP attribute enables neighbor discovery, automatically discovers VTEPs in VXLAN networks, and automatically creates VXLAN tunnels between VTEPs
|
Controller
System-controller
|
|
Tunnel establishment method between VTEPs
|
Adopts a static configuration approach
|
IS-IS protocol extension mode
|
VXLAN tunnels are automatically established through BGP extended attribute associations
|
Controller
System-controller
|
|
Address learning
|
Learns MAC address flooding
|
IS-IS protocol is implemented to learn remote addresses
|
Implementation of extended MP-BGP protocol
|
First packet reported,
The forwarding table item is issued by the Controller
|
|
Standardization
|
No control plane
|
Private protocol, extend IS-IS packets
|
The control protocol of EVPN has been standardized, see RFC 7348RFC7209, RFC7432
|
VTEP and Controller are based on the standard protocol framework Netconf/0penFlow, but each manufacturer has its own private definition, so they are not compatible
|
▲ Table 2 VXLAN control plane technology comparison
From the above table, we can see the benefits of using EVPN as the control plane:
1. Routing Protocols
For medium and large data centers, most of the routing protocols for underlay networks have adopted BGP, and EVPN is implemented through the extended protocol of BGP. Therefore, only one set of routing protocols needs to be maintained, which has more O&M advantages.
2. Standardization
2. Standardization
Compared with other control plane technologies, only EVPN has a standardized definition, which can achieve network heterogeneity and facilitate network compatibility and expansion.
Implementation Solution Based on EVPN+VXLAN
Next, let's take a look at the specific implementation solution based on EVPN+VXLAN. The network architecture is shown in the following figure:
▲ Figure 1 Physical network architecture
Type 3 routing - tunnel establishment
Based on EVPN-Type3 routing - Inclusive Multicast routing1
Automatically discover VTEPs (VXLAN Tunnel End Points, which are the endpoints of VXLAN tunnels) in the VXLAN network, create VXLAN tunnels between VTEPs, and automatically associate VNIs with VXLAN tunnels.
1This type of routing is mainly used for automatic discovery of VTEPs and dynamic establishment of VXLAN tunnels in the VXLAN control plane. VTEPs that are BGP EVPN peers transmit Layer 2 VNI and VTEP IP address information to each other through Inclusive Multicast routing. In particular, the Originating Router's IP Address field is the local VTEP IP address. If the peer VTEP IP address is reachable by Layer 3 routing, a VXLAN tunnel to the peer is established.
EVPN-Type2 Routing ——MAC and Routing Synchronization
Based on EVPN-Type2 routing - MAC/IP routing2, MAC and routing synchronization in the network can be automatically completed.
2The main functions of this type of routing are: host MAC address notification, host ARP notification, and host IP route notification.
EVPN-Type5 Routing —— Border Routing Access
1This type of routing is mainly used for automatic discovery of VTEPs and dynamic establishment of VXLAN tunnels in the VXLAN control plane. VTEPs that are BGP EVPN peers transmit Layer 2 VNI and VTEP IP address information to each other through Inclusive Multicast routing. In particular, the Originating Router's IP Address field is the local VTEP IP address. If the peer VTEP IP address is reachable by Layer 3 routing, a VXLAN tunnel to the peer is established.
EVPN-Type2 Routing ——MAC and Routing Synchronization
Based on EVPN-Type2 routing - MAC/IP routing2, MAC and routing synchronization in the network can be automatically completed.
2The main functions of this type of routing are: host MAC address notification, host ARP notification, and host IP route notification.
EVPN-Type5 Routing —— Border Routing Access
In actual network deployment, we can choose either a Spine as the Border or a group of Leaf nodes as the Border, depending on the traffic in our actual network. If the traffic is mainly north-south, generally a group of Leaf nodes can be used as the Border. However, in some scenarios, such as hybrid cloud, east-west traffic still needs to be forwarded by the Border, so we recommend using Spine as the Border.
In this scenario where Underlay and Overlay are interconnected, a relatively important routing type of EVPN will be used - segment routing, which is transmitted through EVPN-Type5 routing - IP prefix routing. When the external route enters the Border, the Border will be synchronized to the VXLAN network through Type5 routing, thereby enabling hosts in the VXLAN network to access the external network.
In this scenario where Underlay and Overlay are interconnected, a relatively important routing type of EVPN will be used - segment routing, which is transmitted through EVPN-Type5 routing - IP prefix routing. When the external route enters the Border, the Border will be synchronized to the VXLAN network through Type5 routing, thereby enabling hosts in the VXLAN network to access the external network.
Optimization of Distributed VXLAN Solution Deployment
1. Optimization point 1: ARP flood suppression
When the scale of virtual machines in the network expands, ARP broadcast will consume network bandwidth and there will be a risk of broadcast storm. In order to reduce the impact of ARP broadcast, we can suppress ARP flooding through ARP proxy.
ARP proxy implementation: After the gateway turns on the ARP proxy, the host learns that the remote ARP is the MAC address of the gateway. It has the following advantages:
When the scale of virtual machines in the network expands, ARP broadcast will consume network bandwidth and there will be a risk of broadcast storm. In order to reduce the impact of ARP broadcast, we can suppress ARP flooding through ARP proxy.
ARP proxy implementation: After the gateway turns on the ARP proxy, the host learns that the remote ARP is the MAC address of the gateway. It has the following advantages:
●
Reduce the broadcast domain and suppress broadcast storms;
● Increased cluster size: Since the ARP proxy is enabled, the Layer 2 broadcast becomes a Layer 3 route, so each VTEP only needs to maintain its own host MAC and can turn off the remote MAC synchronization function, thereby saving MAC table space and providing support for a larger cluster size.
However, it may not be applicable in some scenarios. For example, the DR mode of LVS requires LVS to learn the real MAC address of the server, so it cannot be replaced by the gateway MAC. In this scenario, we can disable the ARP proxy function of the VNI where the LVS is located to ensure normal communication in the DR mode.
2. Optimization point 2: ARP-TO-HOST
However, it may not be applicable in some scenarios. For example, the DR mode of LVS requires LVS to learn the real MAC address of the server, so it cannot be replaced by the gateway MAC. In this scenario, we can disable the ARP proxy function of the VNI where the LVS is located to ensure normal communication in the DR mode.
2. Optimization point 2: ARP-TO-HOST
▲ Figure 2 Service model
As shown in the figure above, assuming that VM1-4 belongs to the same subnet, this situation may occur due to business deployment or virtual machine migration. If VETP1 and VTEP4 publish network segment routes, due to the existence of ECMP, some traffic to VM1 will bypass VTEP1->VTEP4.
In order to solve this problem, detailed routing or host routing is needed. If you want to configure it manually, the workload is large and it is easy to make mistakes. Currently, a common method is to automatically generate host routing through ARP and publish it to the network.
Implementation method: After the gateway turns on ARP-TO-HOST, the host route is automatically generated according to the ARP table and automatically published to the network.
In order to solve this problem, detailed routing or host routing is needed. If you want to configure it manually, the workload is large and it is easy to make mistakes. Currently, a common method is to automatically generate host routing through ARP and publish it to the network.
Implementation method: After the gateway turns on ARP-TO-HOST, the host route is automatically generated according to the ARP table and automatically published to the network.
Summary
This article explains the implementation solution based on EVPN+VXLAN from the aspects of why distributed VXLAN is used, why EVPN is used as the control plane of VXLAN, the EVPN+VXLAN implementation solution and solution deployment optimization, which is also the recommended solution for Ruijie Network's virtualized data center network architecture. Welcome to leave a message for further communication.
Related Blog:
Exploration of Data Center Automated Operation and Maintenance Technology: Zero Configuration of Switches
Technology Feast | How to De-Stack Data Center Network Architecture
Technology Feast | A Brief Discussion on 100G Optical Modules in Data Centers
Research on the Application of Equal Cost Multi-Path (ECMP) Technology in Data Center Networks
Technology Feast | How to build a lossless network for RDMA
Featured blogs
- Technology Feast | Ruijie Data Center Switch ACL Service TCAM Resource Evaluation Guide
- Silicon Photonics Illuminates the Path to Sustainable Development for Data Centre Networks
- How CXL Technology Solves Memory Problems in Data Centres (Part 1)
- CXL 3.0: Solving New Memory Problems in Data Centres (Part 2)
