In my previous blog I described the concept of the Cisco ACI Policy Model in high level. This blog will take you a bit deeper inside the ACI fabric to understand some of the key ACI constructs and traffic forwarding logic.
In order to understand how the forwarding logic works, first you need to understand the building blocks that construct a Tenant in Cisco ACI.
Building Blocks of a Tenant in Cisco ACI
Although the exact use and definition of a tenant in today’s modern data center networks can vary from solution to solution, in general the concept still the same, which is a logical or physical entity reside within a data center network to serve a certain user group (such as marketing team tenant, guest tenant, test & development tenant etc.). In a modern multi-tenant data center network multiple tenants normally utilize the same underlying infrastructure with logical separation on higher layers (layer 2 to 7).
In Cisco ACI, a tenant can be defined as is a logical container or a folder for application policies. It can represent an actual tenant, an organization, or a domain, or can just be used for the convenience of organizing information.
In other words, in Cisco ACI, typically all application configurations are part of a tenant as it represents a unit of isolation from a policy perspective, and not representing a private network.
Technically, within a tenant, you can define the following:
The concept of a VRF is known and not new even if you are not familiar with the Cisco ACI architecture, a context in ACI is equivalent to a VRF instance. In the APIC GUI, a context is called a “Private Network.” A context defines a Layer 3 address domain, therefore, all of the endpoints within a context /VRF must have unique IP addresses. In addition, if you want one or more subnets to be leaked/shared between different contexts/VRFs then each of these subnet must be unique among the different VRFs.
The figure below illustrates the ACI objects’ structure within a tenant.
What is a “Bridge Domain” ?
Technically, a bridge domain BD is a container for subnets and has impact on forwarding behavior. Keep in mind a bridge domain is not a VLAN, although it can act similar to a VLAN; you instead should think of it as a distributed switch, on a leaf, can be translated locally as a VLAN with local significance. In addition, a DB an act as a broadcast or flooding domain only if broadcast or flooding is enabled, however, this is rarely needed. (we will see an example later in this post when you need to enable flooding in a BD).
Unlink traditional networks where typically traffic is flooded with the Bridge Domain scope (within a VLAN), ACI offers the ability to control the flooding allowance of the following traffic types “as required”.
ACI provide forwarding based on either destination IP or MAC address. The Forwarding logic can be either based on a local table for the directly attached networks/hosts or based global entries in a table that is a cached portion of the full global table that reside in the spine switches. The full global table at the spine switches will be used when an endpoint is not found in the local cache this is also known as Hardware proxy.
By disabling the ARP flooding you will be able to have more efficient forwarding, as ARP/GARP is forwarded as a unicast packet within the fabric based on the host DB.
On the other hand, as it shown in the figure below, at the egress point toward the external L2 switch, the ARP/GARP is forwarded as a flooded frame, this helps to support hosts reachable via downstream L2 switches.
Similarly, designation hosts MAC or IP that are not known by a specific ingress leaf switch are forwarded to one of the spine proxies for address lookup.
With this forwarding logic, if there is no entry present (traffic miss) for that destination in the mapping database in the spine, the packet is handled differently for bridged versus routed traffic as following (1):
The only obvious advantage that someone may think of , when disabling hardware-based proxy and using flooding for unknown hosts and ARP is that the fabric does not need to learn millions of source IP addresses coming from a given port. A BD can be defined as either:
From configurations point of view, because the bridge domain is a child of the VRF instance, so even if you need a purely Layer 2 network, you still need to create a VRF instance and associate the bridge domain with that VRF instance. The good thing here is that if you don’t enable routing, no VRF resources will be allocated in hardware for that VRF. Also, whenever you create an EPG, you will need to reference a BD.
Based on that, the relationships among the various objects can be summarized as follows:
With this approach by grouping EPGs into application profile the administrator makes the network aware of the relationship among application components regardless of where this application physically connected to the fabric ( physical or virtual). The figure below illustrates the relationship among the aforementioned building blocks of a tenant.
First Hop Default Gateway in ACI
At this stage you know how the ACI forwarding construct is structured within a tenant with regard to BDs, VRFs, EPGs and ANPs. At this stage, you need to understand how the first hope layer 3 gateway can be defined and works in Cisco ACI to route traffic between the different subnets within a tenant.
The good news here is that you don’t not need to worry about configuring additional protocol to provide layer 3 gateway services such as HSRP, because Cisco ACI offers what is known as “Pervasive SVI” that provides a distributed default gateway (anycast gateway), which is global across the fabric and configured on top-of-rack (ToR) leaf switches wherever the bridge domain of a tenant is present with unicast routing enabled. In addition, the subnet default gateway addresses are programmed in all Leaves with end points present for the specific tenant IP subnet. This not only helps to reduce design and operation complexity, it helps to large extent to optimize traffic forwarding across the ACI fabric, because when a packet needs to be routed to another leaf or external network, it dose not need to span the network to reach its default gateway that could reside on another leaf, as illustrated in the figure below.
That said, Cisco ACI still support external gateway, which is commonly used when the Fabric is deployed to provide layer 2 transport only for a specific tenant, or if the organization security standards dedicate that the default gateway must be a firewall. With this approach you will need to enable ARP flooding at the BD without enabling unicast routing.