Choosing Your Next Data Center Fabric: Cisco’s ACI or EVPN, Part 1

Cisco customers have a choice between Application Centric Infrastructure (ACI) and Ethernet VPN (EVPN) for building their data center network fabrics. Since these technologies are thought to be interchangeable, the choice often falls to cultural, political or other “Layer 8” decision-making. But the fabric technologies are, in fact, different in ways both subtle and significant, and one’s choice has a profound impact on fabric orchestration, operations, and staffing.

Learn how CDW services and solutions can help you effectively deploy network solutions from Cisco.

I hope you’ll join me all the way through this three-part blog series as I compare and contrast these technologies and provide a foundation for deciding on the best choice for your organization’s needs.

Background on Network Fabrics

A network fabric is a collection of network switches that behave as one, which typically means:

• Any port of any switch can be configured for any Virtual Routing and Forwarding (VRF), Layer 3 subnet, and/or Layer 2 VLAN.
• Forwarding happens as close to the host as possible.
• Host-facing links operate in the overlay and follow traditional networking rules. Fabric-internal links form the underlay and are completely independent of and invisible to the overlay. For example, fabric links can be added/removed and fabric-internal switches can be taken offline, all without concern for the overlay.
• A fabric manager or controller may be used to maintain the overlay/underlay configurations.
• Although fabrics can be built from many topologies, the most common is spine/leaf (diagram follows).

With this topology, the spines and leaf count can be independently scaled to achieve the desired footprint and oversubscription rate. Wire-rate fabrics are possible. All northbound links attach to border leaves.

What Is ACI?

Cisco’s Application Centric Infrastructure (ACI) is a proprietary turnkey fabric introduced in 2014 that works with Cisco Nexus 9300/9500 series switches in ACI mode and uses Virtual Extensible LAN (VXLAN) tunnels. ACI includes the Application Policy Infrastructure Controller (APIC) as the single point of configuration, management and API access. ACI offers many bells and whistles such as policy (security), but this blog series focuses on its management and forwarding features.

What Is EVPN?

Ethernet VPN (EVPN) is a 2015 IETF standard that defines Layer 2 forwarding over VXLAN and Virtual Private LAN Service (VPLS) tunnels using Border Gateway Protocol (BGP) as a control plane. EVPN is a standards-based way to implement a fabric that is functionally similar to ACI. EVPN works on the Cisco Nexus 9300/9500 in NX/OS mode, but it has also been adopted on other Cisco platforms, as well as on switches from Arista, Juniper and others. Cisco’s Data Center Network Manager (DCNM) is optional software used to orchestrate and manage an EVPN fabric, similar to the role APIC plays with ACI.

What About SDN?

With SDN, a controller is closely tied into the network’s control and data planes and the network does not function without it. This architecture is still used in software like VMware NSX-T, but it never took off in hardware. (BigSwitch was the only major hardware SDN solution, but it’s been mothballed since its acquisition by Arista.) Instead, the networking industry has pivoted to orchestration. For example, Cisco’s APIC/DCNM, Arista’s CloudVision Portal (CVP) and Juniper’s Contrail are essentially power tools for modeling and configuring the network and, should those tools go offline for some reason, the network continues to operate with little impact.

Do I Need a Fabric?

Absolutely not! Network operators have long built data center networking from Multi-chassis Link Aggregation Group (MLAG) technologies like Cisco Virtual PortChannel (VPC), and that’s often still the most appropriate solution. However, a fabric is likely in your future if you:

• have more than a handful of switches
• want easier, modern orchestration
• are using Fabric Extender (FEX), Overlay Transport Virtualization (OTV), Virtual Private LAN Service (VPLS), Multi-Protocol Label Switching (MPLS), FabricPath or other features that are vanishing from Cisco’s portfolio

The Value of Standards-based versus Proprietary

When it comes to network fabric, it’s common sense that a standards-based approach would be preferred over a proprietary approach, but that’s not always the case. Consider WAN, where enterprises have happily jettisoned their standards-based MPLS and IPSec VPN networks in favor of proprietary, non-interoperable SD-WAN solutions from Viptela, SilverPeak, Meraki, VeloCloud, CloudGenix, etc.

One reason for this is that standards bodies focus on control and data planes and lowest common denominator features, while most enterprises are more interested on the management and orchestration planes, welcoming any innovation that makes solutions easier, more intuitive and more cloud-like to operate.

Another reason is feature velocity, where proprietary solutions more quickly react to market demand. For example, multicast routing is a basic networking feature that ACI has supported since 2016 but that, as of 2021, EVPN still has no standard for. The standards process has taken so long that vendors have come up with their own approaches. Cisco introduced Tenant Routed Multicast in 2017 (now draft standard MVPN). Meanwhile, Juniper came up with a different approach (draft standard OISM, recently adopted by Arista). This is not the first or last time that standards bodies have led to fragmentation, stagnation and confusion.

With EVPN, there are a few areas where standards-based can be important:

1. Multi-vendor within the fabric: As one might need when migrating between vendors or when different teams manage different parts of the fabric. Multi-vendor fabrics are exceedingly rare and will have trouble with vendor-specific features like multicast and VPC. Apstra, recently purchased by Juniper, is the only fabric manager that claims support for multiple vendors in the same fabric.
2. Portability of engineering skills: An engineer who understands EVPN internals on Cisco will be very fluent with EVPN on Arista or Juniper.
3. When standards are superior: Although standards typically lag, there are refreshing exceptions. One is EVPN multihoming, a standard that can eliminate the need for proprietary multi-chassis link aggregation like Cisco VPC, Arista MLAG, Juniper MC-LAG and Avaya SMLT.

Separately, EVPN may be preferred simply so one can take advantage of NX/OS mode benefits:

• Standards-based automation – Yet Another Next Generation (YANG), Google Remote Procedure Call (GRPC), OpenConfig
• Support for existing operational tooling – Simple Network Management Protocol (SNMP), Command Line Interface (CLI), NXAPI
• Familiarity

If NX/OS mode is the main driver for choosing EVPN, then the organization may be planning to use their own automation rather than using Cisco’s DCNM fabric manager.

Cost Comparisons between ACI and EVPN

There is little cost difference between Cisco ACI and EVPN since they mostly run on the same hardware and have similar licensing. The following table shows how much more (or less) ACI costs over EVPN.

For example, if a 16-leaf EVPN fabric + subscription was quoted at $500,000, the ACI equivalent would cost 3.6 percent less, or $482,000. This comparison model includes the cost of APIC controllers for ACI, but not the compute cost for the EVPN’s optional DCNM fabric manager. It also compares ACI’s entry-level Essentials license with NX/OS’s more expensive Advantage license, since that is required for feature parity with multicast routing and enhanced policy-based-routing. This also means the compared EVPN prices include the Multi-state featured, described below.

Where EVPN could lower costs is with very small deployments that can be engineered with no spine or a very small spine. However, such small networks would likely be better served by a traditional NX/OS VPC design rather using than EVPN.

There are a few notable pricing details that customers should be aware of. First, spine switches require no licensing with ACI, but they do with EVPN. Second, entry-level ACI includes Multi-Pod, a powerful feature that allows up to 12 pods and 500 switches within 2000 miles to be orchestrated as a single distributed, survivable fabric so long as the pods are interconnected with 10+ Gbps links. For many enterprises, ACI Multi-Pod is a slam dunk multi-DC solution that is much simpler than ACI Multi-Site. EVPN has no Multi-Pod feature, but its Multi-Site feature is similarly attractive and is included in this price comparison.

One minor cost difference is with the choice of spine hardware. Many ACI fabrics today are built with Nexus 9332C or 9364C spines, which are pure underlay switches at a low price point. EVPN can also use these, but only in a pure underlay mode. In practice, many EVPN architectures are not pure and require the spines to handle both underlay and overlay networking, and this requires more expensive FX or GX spines. This may be a moot point by 2022, as Cisco’s latest GX2 spines are equally capable of both ACI and EVPN support.

Blog Series Links

This blog series continues in Choosing Your Next Data Center Fabric: Cisco’s ACI or EVPN, Part 2 where I will delve into the technical differences between these two technologies, and then in Choosing Your Next Data Center Fabric: Cisco’s ACI or EVPN, Part 3, where I will be focusing on what differentiates these two technologies to help you decide which one is right for you.