Community discussions

MikroTik App
User avatar
Topic Author
Posts: 31
Joined: Sat Nov 21, 2020 1:25 am

ISP network design using switches

Mon Aug 15, 2022 3:40 am

In order to simplify my network, I would like to make some changes. Based on the following design guide, I am currently running OSPF and BGP:

My network consists of 15 towers and one edge router that connects to the internet. Each tower has redundant links to the edge router. In my diagram, I used three towers to simplify my explanation.

By designing the network as follows, I may be able to use only bgp:
  • A vlan should be tagged on all switches that Tower3 uses to connect to the Main router.
    Tower3 will see Main router as next hop, so if Route1 is unreachable, it will automatically switch to alternate route. This eliminates the need to use Recursive Routing, Netwatch, or other internet monitoring methods for the gateway.
  • A vlan will be tagged from Tower3 to Tower2 and from Tower3 to Tower1, in order to access any services that are hosted on the respective towers. For example, if Tower1 hosts a FTP server, Tower3 doesn't have to go all the way to the main router and back to Tower1.
The edge router can act as a Route Reflector for my BGP config, but will it allow adjacent towers to connect directly with each other or will it re-advertise routes in such a way that traffic is routed through the edge router and then back to the target tower?

Here is a simplified diagram:
Anybody else implemented something like this? If so, how's it going?
You do not have the required permissions to view the files attached to this post.
Frequent Visitor
Frequent Visitor
Posts: 50
Joined: Sun Apr 28, 2019 9:01 pm

Re: ISP network design using switches

Sat Aug 20, 2022 6:17 am

Whats the advantage of using over BGP only vs hybrid OSPF/BGP? Less complexity? It seems more like a L2 network with BGP on top....Im not trying to knock it...just dont fully grasp it yet.
just joined
Posts: 14
Joined: Tue Apr 13, 2021 4:40 pm

Re: ISP network design using switches

Sat Aug 20, 2022 7:17 am

If I'm understanding correctly, you're trying to replace some L3 functions with L2 functions? I'd recommend you route between sites, and trunk within sites. The OSPF protocol can easily handle all of the pathing without you needing to manually configure any routes at all, I'm not sure why you would want to use BGP within your own boarder either.

My vote is tighten up your OSPF configuration and make it do all the work for you.
Member Candidate
Member Candidate
Posts: 122
Joined: Tue Dec 19, 2017 9:45 am

Re: ISP network design using switches

Sat Aug 20, 2022 5:48 pm

I stared at this a while until I think I got my head around why this might look preferable to you.

It seems like rather than accepting the physical ring shape of your network, the switches would enable somewhat of a transport medium for a series of “point-to-point” circuits between routers that otherwise can’t have direct links (like tower 3 to the internet edge router).

This isn’t an uncommon concept but you typically see it implemented at the optical layer with circuit protection going opposite directions around the ring. I’m not suggesting you should do that, just observing that it’s not unheard of to want to convert a physical ring into a full mesh.

There are some big differences between doing this at the optical layer and doing it with Ethernet tagging, though.

A big one is contention. Implemented in the optical domain, there is no traffic contention between lambdas on the ring. Implemented as a tunnel, routers are still in contention with each other for bandwidth and sometimes in unintuitive ways. This isn’t worse than routing hop-by-hop around the ring, but it’s not better either. All it’s done is hidden the true path for troubleshooting purposes. You’re still going to have to inspect the same physical interfaces around the ring as you would have if you just routed hop-by-hop.

Another comparison is in terms of resulting link complexity. Implemented in the optical domain, there is only one resulting circuit and so the resulting router topology has less links than there are physical paths. Implemented with switching, there is still a need for two routed paths. Whether that is around the ring in both directions to the internet edge or just two routed links to the two towers, this hasn’t reduced the links in the routing process for a given path.

All in all, I don’t see what it specifically gets you in exchange for more opaque troubleshooting.

Additionally, because these VLANs end up being point-to-point in nature, you will still have a nontrivial topology inside the network. Even though each tower would be able to reach the route reflector, they won’t have a clue how to reach each other. BGP carries no path information inside an AS.

This is a problem because, for example, tower 1 and tower 2 would learn each other’s client routes via BGP, but the next hop would be either a loopback address or an interface address that it wouldn’t know how to reach. As a result, it wouldn’t install them in the FIB. The situation can be ”fixed” by setting next-hop self for those routes, but that just forces traffic to always take the path through the route-reflector. No matter what you do, tower 1 and tower 2 still don’t know how to reach each other through tower 3 and you have lost path redundancy that you had at the start.

With four routers in a ring I would just run OSPF on the physical interfaces and an iBGP mesh between loopbacks and call it a day. Anything else makes the network harder to configure and troubleshoot for no substantive technical gain.

Who is online

Users browsing this forum: No registered users and 5 guests