BGP PIC Fundamentals: BGP PIC (Prefix Independent Convergence) is a BGP Fast Reroute mechanism which can provide sub-second convergence even for the 900K internet prefixes by taking the help of IGP convergence.
BGP PIC uses a hierarchical data plane in contrast to a flat FIB (Forwarding table) design which is used by Cisco CEF and many legacy platforms. In a hierarchical data plane, the FIB used by the packet processing engine reflects recursions between the routes. I will explain the recursion concept throughout the post, so don't worry about the above sentence; it will make sense.
There are two implementations of the BGP PIC concept, and they can protect the network traffic from multiple failures. A link or a node in the core or edge of the network can be recovered under a second, and in most cases under 100ms (it mostly depends on IGP convergence, so IGP should be tuned or IGP FRR can be used). In this article, I will not explain IGP fast convergence or IGP Fast Reroute mechanisms in detail.
BGP PIC can be thought of as a BGP Fast Reroute mechanism that relies on IGP convergence for failure detection. (All overlay protocols rely on underlay protocol convergence, i.e. LDP/IGP Synchronization, STP/HSRP, IGP/BGP, IP/GRE, and so on.) I am explaining this topic in deep detail in the BGP Zero to Hero course.
As I mentioned above, there are two implementations of BGP PIC, namely BGP PIC Edge and BGP PIC Core. Let's start with BGP PIC Core.
BGP PIC CORE
Consider a topology where R1, R2, R3, R4, and R5 belong to AS 100, and R6 and R7 belong to AS 200. There are two EBGP connections between the ASBRs of the Service Providers.
Everybody told you so far that BGP is slow because BGP is used for scalability in the networks, not for fast convergence, right? But that is wrong too. Or at least not enough to understand how BGP converges!
If BGP relies on the control plane to converge, of course it will be slow, since the default timers are long (BGP MRAI, BGP Scanner, and so on, although you don't need to rely on them as I will explain now), and prefixes and path information are too much for the Best Path Selection algorithm to select the second-best path to advertise in case the primary path fails.
The default-free zone already has more than 900K prefixes. So approximately we are talking about 100 MB of data from each neighbor, and it takes time too. If you have multiple paths, the amount of data that needs to be sent will be much higher.
Let's look at BGP control plane convergence closer.
Imagine that R1 in this topology learns the 5.5.5.5 prefix from R4 only. R4 is the next hop. (You choose maybe R4 as a primary link with BGP Local Preference or MED, or you don't do anything but R4 is selected by hot potato routing because of the Route Reflector position.)
If R4 is the best path, R5 doesn't send the 5.5.5.5 prefix to the IBGP domain unless BGP best external is enabled (I highly recommend you to enable it if you want additional path information in the Active/Standby link).
How did IBGP routers learn that R4 failed? There are two mechanisms for that. They will either wait for the BGP Scanner time (60 seconds in most implementations) to check if the BGP next hop for the BGP prefixes is still up, or the newer approach, BGP Next Hop Tracking (almost all vendors support it). With BGP next-hop tracking, BGP next hop prefixes are registered to the IGP route watch process, so as soon as IGP detects the BGP next-hop failure, BGP is informed. It is similar to BGP, IGP, and LDP registration to the BFD, right? Good!
So R1 learned the R4 failure through IGP. Then R1 has to go and delete all the BGP prefixes which are learned from that next hop. If it is a full internet routing table, it is a very time-consuming process as you can imagine. I am talking here about minutes.
In the absence of an already calculated backup path, BGP will rely on this control plane convergence, so of course it will take time. But you don't have to rely on that. I recommended many service providers start to consider BGP PIC and Egress FRR for their Internet and VPN services.
In the router's routing table, there is always a recursion for the BGP prefixes. So for the 5.5.5.5 prefix, the next hop would be 10.0.0.1 if next-hop-self is enabled. But in order to forward the traffic, the router needs to resolve the immediate next hop and the Layer 2 encapsulation, if it is an Ethernet MAC address.
For the BGP next-hop 10.0.0.1, R1 selects either 172.16.0.1 or 172.16.1.1 as an IGP next hop. Or R1 can do ECMP (Equal Cost Multipath) and thus can use both 172.16.0.1 and 172.16.1.1 to reach 10.0.0.1.
In many vendor FIB implementations, BGP prefixes resolve the immediate IGP next hop. Cisco's CEF implementation works in this way too. This is not necessarily a bad thing though. It provides better throughput since the router doesn't have to do a double/aggregate lookup. But from the fast convergence point of view, we need a hierarchical data plane (Hierarchical FIB).
With BGP PIC, both the PIC Core and PIC Edge solutions, you will have a hierarchical data plane, so for the 5.5.5.5 prefix you will have 10.0.0.1 or 10.0.0.2 as the next hop in the FIB (same as the RIB). For 10.0.0.1 and 10.0.0.2, you will have another FIB entry that points to the IGP next-hops, which are 172.16.0.1 and 172.16.1.1. These IGP next-hops can be used in a load-shared or active/standby manner.
BGP PIC Core helps to hide IGP failure from the BGP process. If the links between R1-R2 or R2-R3 fail, or R2 or R3 fails, R1 will start to use the backup IGP next-hop immediately. Since the BGP next-hop didn't change and only the IGP path changed, recovery time will be based on IGP convergence.
For BGP PIC Core, you don't have to have multiple IBGP next hops. BGP PIC Core can handle core IGP link and node failure.
BGP PIC EDGE
Let me explain BGP PIC Edge, which can handle edge link or node failure in a slightly different way than BGP PIC Core for some scenarios. BGP PIC Edge provides sub-second convergence time in the case of edge link or node failure — sub-second convergence is not possible for BGP without PIC (Prefix Independent Convergence).
In order for BGP PIC Edge to work, edge IBGP devices (Ingress PEs and ASBRs) need to support BGP PIC, and they also need to receive a backup BGP next hop.
Unfortunately, the backup next hop is not sent in IBGP Route Reflector topologies, since the RR selects and advertises only the best path for a given prefix. For example, if R6 and R7 both send the same network, R1 can learn only one exit point (BGP next-hop) from the RR — either R4 or R5. Note also that placing the Route Reflector in the data path is not recommended in real network design.
Another drawback of the Route Reflector is that when it needs to do hot potato routing by calculating the IGP cost to the BGP next-hop, it takes only its own cost to the next hop into consideration. The Route Reflector's IGP cost calculation to the BGP next-hops might be different from the Ingress PE's cost calculation to the same next-hops. Thus the Route Reflector may not provide an optimal path for all the Ingress PEs. The BGP Optimal Route Reflection draft specifies a couple of solutions for this problem.
How would you send more than one best path from the Route Reflector to the Route Reflector clients? There are many ways to do it, but the two famous ones are BGP Add-Path and BGP Diverse Paths (multiple control plane RRs).
Assume now we have more than one path on R1: R1 learns the prefix from both R4 and R5 and chooses R4 as the best exit point. In this case, R5 is marked as the backup/alternate next hop and programmed into the FIB (Forwarding Information Base) of R1. Let's examine the edge failure scenarios and see how BGP PIC Edge takes action.
Edge Node Failure When the ASBR Sets Next-Hop-Self
In the first case, we are doing BGP next-hop-self on R4 and R4 fails. This failure information is detected by IGP, and next-hop tracking removes the BGP next hop from the BGP path list on R1. The alternate backup route can be used immediately — R1 changes the next hop to R5 for all the BGP prefixes.
This is BGP data plane convergence, not control plane convergence, so the convergence time is only related to IGP convergence and is prefix independent. If you have a 500K full internet routing table, all of the prefixes will be installed in the FIB before the failure as backup routes, and when the failure happens, the next BGP next hop is used immediately. Convergence is sub-second regardless of the number of prefixes.
Edge Link Failure When the ASBR Sets Next-Hop-Self
The second failure scenario might be the edge link between R4 and R6, while R4 is our primary next-hop and we are doing next-hop-self on R4 (in MPLS VPN, you always do that!).
When R4 sets the BGP next hop to itself (done by setting the loopback as the next hop), the loopback interface won't go down when the edge link fails. Even though R1 may learn of the link failure from the IGP (if the R4-R6 link is redistributed into the IGP), it doesn't trigger BGP next-hop tracking to fail the BGP next hop, because the BGP next hop for the prefix is not the R4-R6 link address but the R4 loopback. Since BGP's next hop doesn't change on R1, R1 continues to forward the traffic according to the IBGP best path selection sent by the RR towards R4.
In this case, R4 should redirect the packets to its alternate second-best path, which is R5. That's why not only the ingress nodes but also the ASBRs should learn the alternate next hop. But in an IP environment without tunneling (GRE, L2TPv3, IPinIP) or encapsulation (MPLS), intermediate nodes which are not converged yet would send the packet back to R4, since they would think that R4 is still reachable, so it would be a temporary loop. In the case of MPLS or other tunneling mechanisms, intermediate nodes wouldn't need BGP, so they would just send packets to the second-best path as per the R4 request.
Edge Link Failure When the ASBR Does Not Set Next-Hop-Self
What if the R4-R6 link fails and R4 doesn't set the next hop to itself (no next-hop-self)? In that case, the link between R4 and R6 is advertised into the IGP. When the R4-R6 link fails, R1 learns the failure from the IGP, and the BGP next-hop tracking feature helps here: IGP protocols register to the BGP next-hop tracking process (similar to the IGP-to-BFD registration, right?).
When R1 learns of the link failure between R4 and R6, it immediately changes the BGP next hop for the affected prefixes to R5. This switchover is done in less than a second regardless of the number of prefixes — even if you have millions of BGP prefixes which need to be updated, convergence is still sub-second.
When Do You Set Next-Hop-Self on the Edge BGP Node?
In an MPLS VPN environment, it is best practice and almost mandatory to set the BGP next hop to the PE loopback. With that, the transport LSP can be created to that PE loopback.
BGP PIC Key Takeaways
- BGP PIC (Prefix Independent Convergence) is a BGP data plane convergence mechanism, not a control plane one. Convergence time is only related to IGP convergence and is prefix independent.
- BGP PIC is also known as BGP Fast Reroute.
- BGP PIC is very suitable for MPLS VPN service, especially when the customer pays for and requires tight convergence times for any failure. It can be provided by the Service Provider as a value-added service and thus might provide additional revenue.
- When BGP PIC is enabled, adding a backup path for millions of prefixes requires additional memory and CPU; that's why resources should be monitored carefully.
- BGP PIC is not necessarily only a BGP feature, since BGP simply takes advantage of recursion and the hierarchical data plane arrangement. It is also not a Cisco proprietary mechanism — most of the vendors, including Juniper, implement BGP PIC today.
