I'm also really confused as to why they insisted on taking full tables at their edge if they also built their own route optimization technology. It seems like it would be much simpler and faster if they just took a default route from each provider and then did route probing on their top 30% prefixes in each PoP. You don't need full edge routes for that.
You need a single route reflector (or two for redundancy) to tell you what routes are in the table. Then probe the busiest prefixes (as determined by flow data) via each provider and take the best route. Localpref the best route higher in the route reflector and send it to the Linux hosts. No need for a huge FIB on the edge. This is basically what Noction IRP/Internap FCP platforms already do.
This reminds me of the scene in Primer where they ask if they're doing it for a reason or just showing off.
FWIW, the Netflix networks just take a default route from a pair of transit networks in each PoP. At their scale, they found it better to negotiate better worldwide transit deals than optimize networking at the PoP level. Different game than Fastly, but another data point.
I remember those FCP boxes, its been a while. I think its maybe what they call Miro now, do you know? The Avaya route science boxes did the same thing. It seems both of these have fallen out of usage. I always thought it was odd that there wasn't an opensource project for this. I know Internap originally did it in PERL.
Yep, FCP's successor is Miro. I haven't had a chance to look into it, but their sales people claim it's all new.
Noction has pretty much taken over the entire market. Overall their product is good, but the Internet game has changed a lot since the original FCP days (lots more direct peers, lots more exchanges, lots more route servers, lots more paid peering), and I don't think that things like FCP/Miro/IRP do well once you move past paid peering. Things just become too complex. IRP exposes some of this to you, but the implementation is kludgey (routing instance per peer, DSCP bits to identify peers on exchanges, and more).
Agreed about how the Internet game has changed. If I remember correctly Internap didn't peer with anybody they simply bought tons of transit and connected the PNAPs with local fiber.
I didn't understand your comment about "... once you move past paid peering."
Did you mean "once you move to paid peering"? I would think that moving past it would be actual settlement-free peering. But I'm curious what you meant.
> It seems like it would be much simpler and faster if they just took a default route from each provider and then did route probing on their top 30% prefixes in each PoP.
Probing just 30% of the top prefixes is likely not enough for their use case. Probing is also an after the fact approach. Fastly needs a solution which will give best possible path from first packet.
> FWIW, the Netflix networks just take a default route from a pair of transit networks in each PoP. At their scale, they found it better to negotiate better worldwide transit deals than optimize networking at the PoP level. Different game than Fastly, but another data point.
Different game indeed. Netflix just cares about throughput, Fastly also needs low latency, hence the need for more optimizing.
"Probing just 30% of the top prefixes is likely not enough for their use case. Probing is also an after the fact approach. Fastly needs a solution which will give best possible path from first packet."
That's not how BGP works. There's no such thing as getting the best path from the first packet in terms of latency. There's nothing in BGP that tell you about the path in terms of latency or performance. The only "best" that BGP gives you is AS path length between you and the destination. It tells you nothing about congestion/packet loss upstream. The only way to get that is out of band probing, like taking latency measurements and pref'ing routes. There is no approach beyond "after the fact"
There is nothing in this article that talks about optimization of routes.
Of course not. I only stated what Fastly needs, I did not make a statement about how BGP works.
By having full BGP feeds from all transit providers the nodes can use the AS paths as part of the heuristics to determine the best possible path (with the available information) from first packet. Obviously the heuristics will also use after the fact probe information to tune the model.
The point is that the solution proposed by the grandparent is not better than the one described in the article.
> There is nothing in this article that talks about optimization of routes.
What you're talking about works OK in a single colo or datacenter model, but it doesn't map to the PoP model. It's the PoP model that's driving them to do this. In their PoPs they've got local peers and exchanges that will handle the bulk of their traffic. The leftover stuff is only local and probably minor. They can definitely find that with flow data and optimize across a pair of default routes (basically building their own routing table). It's a pretty common practice when all you deploy is PoPs. Why do you care about a full BGP feed when you're only handling 4-10% of the Internet in a given PoP?
AS path length doesn't tell you anything about a route's performance. It's a hint and only a hint.
> What you're talking about works OK in a single colo or datacenter model, but it doesn't map to the PoP model. It's the PoP model that's driving them to do this
Could you please explain how the PoP model in this case differs from the colo/DC model?
Outside the US Fastly's PoP cover multiple countries or whole continents.
> Probing just 30% of the top prefixes is likely not enough for their use case. Probing is also an after the fact approach. Fastly needs a solution which will give best possible path from first packet. <
Considering that Fastly deploys PoPs and not colos or datacenters, I think that the top 30% of their transit routes would be the lionshare of non-peer activity. Fastly's PoPs are generally at (or have reach to) the largest Internet exchanges for peering purposes. Peering routes are generally the best, cheapest, most concise routes you can get. You don't need a big, expensive router when you're in an exchange's peering mesh. Considering that you're only handling local traffic (by design of having multiple PoPs), peering plus two default transits will probably optimize for the traffic that matters most.
Also as mentioned, you can't really determine "best possible path" without probing. You can see what BGP tells you, but that's not always the best possible path. People that run networks know this. That's why things like FCP/IRP/RouteScience exist (for better or worse).
> Different game indeed. Netflix just cares about throughput, Fastly also needs low latency, hence the need for more optimizing. <
Sure, but you won't get that optimization from BGP. In a Fastly PoP, you're mainly dealing with in-country or in-region eyeball networks. If you buy transit from 2-3 major networks, BGP probably won't help you with those eyeballs. The path length will probably be the same (the eyeballs either peer or buy transit from large tier 1s). You generally can't trust MED to do anything useful with it, so that's out. Those are the only 2 things BGP can use to impact route selection. If path length and MED are the same, it's a coin flip as to which path to take (not really as the first installed route is the winner).
If you just took default routes and peered locally, you'd be much better off in a PoP scenario than trying to re-invent route reflectors and taking full BGP feeds.
You need a single route reflector (or two for redundancy) to tell you what routes are in the table. Then probe the busiest prefixes (as determined by flow data) via each provider and take the best route. Localpref the best route higher in the route reflector and send it to the Linux hosts. No need for a huge FIB on the edge. This is basically what Noction IRP/Internap FCP platforms already do.
This reminds me of the scene in Primer where they ask if they're doing it for a reason or just showing off.
FWIW, the Netflix networks just take a default route from a pair of transit networks in each PoP. At their scale, they found it better to negotiate better worldwide transit deals than optimize networking at the PoP level. Different game than Fastly, but another data point.