2

In many architectures there are load-balancing mechanisms such as ECMP routing or LAG between various interfaces. This load-balancing is usually per-flow, based on a hash of various criteria, such as source/destination address, protocol, port information, etc. What I'm wondering is, how do you troubleshoot non-deterministic routing situations like this?

For example, in a deterministic routing scenario where traffic takes only one path, there can be cases of packet loss or increased delay on a link, and it's relatively easy to identify this. If this happened in a non-deterministic routing scenario, I imagine identifying problems with a particular member interface could be difficult to discern.

Obviously, in normal situations, there would be no need for this, but I'm wondering if there are any commands on platforms like Cisco IOS (XE, XR, etc.) or JunOS, etc. which could reveal the result of the hashing algorithm, or if anyone has experience with issues like these and has a particular troubleshooting methodology for it.

Thanks

Improve this question
3
  • 2
    "how do you troubleshoot non-deterministic routing situations like this?" The hashing algorithm is deterministic. If it were not deterministic, you could not guarantee that every packet in a flow takes the same path. Commented Jan 8, 2021 at 19:53
  • 2
    "if you don't know the algorithm, it may as well be non-deterministic!" That is a patently false statement. You depend on a single flow following the same path, and that can only happen if the algorithm is deterministic. If it were not then any packet from any flow could follow any path, and that could be very bad. Just because you do not know the algorithm, does not mean anything. There are proprietary algorithms that are secret, yet many people depend on them. For example, Skype, many people and businesses depended on it, but it uses a proprietary algorithm, and dialing is deterministic. Commented Jan 8, 2021 at 21:32
  • Did any answer help you? if so, you should accept the answer so that the question does not keep popping up forever, looking for an answer. Alternatively, you could post and accept your own answer. Commented Dec 23, 2021 at 15:51

3 Answers 3

Reset to default
3

There are some things to test in certain circumstances, but the basic idea is that PAgP or LACP aggregations are between two directly-connected devices, and anything sent from one device to the other ends up on the other device, regardless of the path of the flow.

Cisco has a command to test the interface used for a port channel:

You can determine which interface in the EtherChannel forwards traffic, with the frame distribution policy as a basis. Issue the remote login switch command to log in remotely to the Switch Processor (SP) console in order to make this determination. Then, issue the test etherchannel load-balance interface port-channel number {ip | l4port | mac} [source_ip_add | source_mac_add | source_l4_port] [dest_ip_add | dest_mac_add | dest_l4_port] command.

Cisco and Juniper have various documents that go into some detail about the hashing algorithms for LACP, mostly describing which parameters are used for the algorithm, but there does not seem to be any specific testing like the above command.

Improve this answer
2

I've been using this command on NX-OS, on a set of N3K-3164Q:

switch# show routing hash <srcIP> <dstIP> ip-proto <protocolNumber> <srcPort> <dstPort> vrf <vrfName>

It helped quite a bit to gain insight into an ECMP polarization problem.

Improve this answer
1

For ECMP you can trace the route based on ICMP errors.

TTL is not included in the hashing algorithm, so as long as you keep the fields that are included in the hash consistent you can trace the route by sending packets with increasing TTLs and monitoring the ICMP errors that come back.

Facebook wrote a specialist traceroute tool for use in ECMP scenarios that loops over a range of source ports to exercise different routes and report packet losses for each route at each hop.

https://github.com/facebook/fbtracert

Afaict this won't work for LAGs though, because the L3 hops will be the same for all the paths.

Improve this answer

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.