1

enter image description here

Hello everyone,

I’m currently working on setting up a redundant Layer 2 link using VXLAN over IPsec for a client. The attached diagram illustrates the current network configuration.

My objective is to ensure redundancy for the Layer 2 connection by leveraging VXLAN over IPsec. However, I am facing some issues, and I suspect the problem lies with the configuration of the Dell switches. Before I focus on the switch configuration, I want to make sure that the static routes on the FortiGate devices are set up correctly.

Here’s what I’ve configured so far:

  • On the FortiGate Audran, I have a static route to the 192.168.77.0/24 network via port11 with the gateway set to 192.168.77.1.
  • On the FortiGate Abreuvoir, I have a static route to the 192.168.77.0/24 network via port11 with the gateway set to 192.168.77.2.

(I admit, this feels a bit weird to me, but can't think of any configuration that would be more logical to me).

I’d like to confirm:

  • Are these static route configurations correct for this setup?
  • Should I adjust the routes or add additional configuration to better handle the redundancy
  • Is there any specific best practice when combining VXLAN over IPsec with such a Layer 2 setup?

Any guidance or suggestions would be greatly appreciated. Thank you in advance!

Improve this question
7
  • At first glance: do not use the same LAN subnet on both Fortigate sites, use different ones. You cannot easily override a directly connected subnet with a static route. Yes, that feels weird because it is. ;-) Beware of the encapsulation overhead of both IPsec and VXLAN - the MTU for the redundant link will be greatly reduced unless you can use baby giants. Commented Nov 20, 2024 at 9:30
  • @Zac67 Thanks. Unfortunately, I cannot change the subnet configuration. Both sites must use the same 192.168.77.0/24 subnet, as it is a strict requirement for this setup. Regarding the VXLAN link, I’m aware that it’s not an ideal solution due to the encapsulation overhead and reduced MTU. However, the VXLAN link is intended to be used solely as a backup in case of a failure on the Layer 2 link. Its purpose is to maintain service continuity during the recovery of the primary L2 connection. Given these constraints, do you have any other suggestions? Thanks again. Commented Nov 20, 2024 at 9:39
  • I remember that Fortigates don't like ambiguous subnets and the required workarounds, so good luck with that. I'd at least use distinct subnets between the Fortigates (via loopback, secondary address, ...) but I'm not sure that it'll fly even then. For the MTU, make sure you test well - it's no use to have a somewhat working fallback that the services on top can't use properly. Commented Nov 20, 2024 at 9:51
  • @Zac67 Alright, could you elaborate a bit on what you mean by using "distinct subnets between the Fortigates"? Are you suggesting that even though both sites must retain the same 192.168.77.0/24 subnet for the connected devices, the Fortigates themselves should communicate over a separate subnet (e.g., via secondary IP addresses)? If so, how would this work in terms of routing and avoiding conflicts with the directly connected subnet? Commented Nov 20, 2024 at 10:45
  • @Zac64 Here’s a link to a Fortinet article that demonstrates how the same subnet can be used on both sides: docs.fortinet.com/document/fortigate/7.2.9/administration-guide/… One thing to note, though, is that in the article's network diagram, there is no L2 link between the switches... Commented Nov 20, 2024 at 11:03

2 Answers 2

Reset to default
3

Is there any specific best practice when combining VXLAN over IPsec with such a Layer 2 setup?

Bluntly: Yes, there is a best practice: Do not do this.

Unless your are confident and you are prepared to jump over fences and through fire rings, and your are bold enough to extort compliance from the sysadmin and software people running the end systems and their applications.

Packet Size Problem:

Assuming the internet path will give you 1500bytes of IP payload size from Audran to Aubrevoir...

  • IPSec will use up to 100 bytes of that for ESP and outer IP headers.
  • VXLAN header will use up another 50bytes
    • 20 bytes for an inner IP header,
    • 8 bytes of UDP header,
    • 8 bytes bytes VXLAN header (incl 24bit VXLAN ID). .
    • 14 inner (original) Ethernet

This leaves - possibly - 1350 bytes of IP MTU through the VXLAN-over-IPsec tunnel.

Since this is a L2 construct, none of the devices along the paths (switches, vxlan encapsulating/decapsulating fortigates) can be expected to either "L2-fragment" packets or to signal to the end hosts that they should be using smaller L3 payload sizes. (Basically, there is no such thing as PMTUd at Layer 2 or fragmentation of ethernet frames. Then again, you never know what the Fortigate people come up with... )

You'll have to make sure that all participating systems on this VLAN/subnet are aware of the reduced maximum L3 packet size in all of VLAN 888, and are configured appropriately, including MTU settings of their network interfaces and possibly even data chunk size at the application layer.

And even then: Redundant paths at L2

You will still need to make sure that there is proper loop detection/prevention in the quadrilateral AC-HQ-SW501a - Fortigate(Audran) - VXLAN-o-IPSec - Fortigate(Abreuvoir) - DELL Switch - L2 Link. Be sure to thoroughly understand how switches and fortigates interact in this.

Also: be aware of the consequences of a failure of "L2 Link" and traffic flows within VLAN 888. Can the fortigates and their internet links take the load? How will the end systems in VLAN 888 deal with a network path that just changed characteristics abruptly (Latency, Jitter, Loss) ?

If management wants to force this upon you, have them sign a waiver absolving you from any consequences, for an element of questional stability they make you introduce to VLAN 888 and possibly the entire network.

Suggestion:

Review and question deeply and thoroughly (applicaton of spanish inquisition practices optional) the true and actual need for end systems 192.168.77.11 and 192.168.77.10 to be in the same broadcast domain.

More often than not, a requirement like this is "just given", but the people requiring it could just not be bothered (or their product resp IP stack is so broken or so outdated it cannot route). In that case: see waiver-from-management, above.

For the classic case of hard-coded or licence-bound IP addresses in the applications, there's better tricks which give you the freedom and stability of routing. For example: Add the hard-coded IP addresses to the host as 2ndary IP or loopback with /32, bind server/listener and client to said 2ndary IP, and eventually add & redistribute static routes on the network accordingly ... but that's already touching the limits of on-topic-ness of this board.

Improve this answer
1

You could enable Proxy ARP on the fortigates,then very carefully set up some static host routes on them: you'd tell audran that the route to TEST-CRAPTRAP (or whatever that says in the lower right) is over the VXLAN link's next hop (abreuvior). And then you'd tell abreuvior that the route to 192.168.77.11 is over the VXLAN link's next hop (audran).

Now you have a race condition. When 192.168.77.11 ARPs for TEST-CRAPTRAP, TEST-CRAPTRAP should answer first, assuming the L2 Link is up and not congested. The frames will stay in Layer 2 and will not hit the fortigates.

When the the L2 Link is down, then the audran will answer the ARP request. Thanks to the heinous host routes on audran, it wll will forward the frames over the VXLAN link to abreuvior, which will forward to TEST-CRAPTRAP.

This is a terrible idea, though. I would not want to be responsible for admining this network.

Improve this answer
0

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.