HSRP router using interface MAC in responses

[mmistrettammistretta]

I have an issue where data being routed through a 6500 sup720 is using its interface mac as the source, rather than the virtual.

ARP to the virtual IP respond with the virtual MAC, however.

ICMP request originating from network 5 (VLAN5) are routed to network 111, and once the ethernet header leaves the 6500, it has the interface mac. The reply, coming into the 6500, is now using the interface mac, rather than the HSRP virtual.

Redirects are disabled on the HSRP interfaces.
This is all part of a much bigger issue that we are experiencing, but this is a clue to why we may be having this problem.


Thanks,
Mike

[Kevin Dorrell]

Mike,

This is normal behaviour in HSRP. When you ARP for the router virtual address, you get back the virtual MAC, for example 00:00:0c:07:ac:01 for group 1. So, for traffic to the router, a host will use the virtual MAC address as destination. But traffic coming back from the the router, (i.e. traffic that the router is forwarding to you) will have the router's built-in MAC address as source.

Now this is interesting, because it begs the question about what keeps the CAM table entries alive for the virtual MAC address. I didn't know this, so I got out my Wireshark and investigated. It seems that the HSRP keepalives themselves do the job. The HSRP keepalive from the active router will have the virtual MAC address, 00:00:0c:07:ac:01, as source. The HSRP keepalive from the standby router will have its own built-in MAC address as source.

Thanks for your question ... I learned something today.
Tell us about the "bigger issues" you mentioned ... maybe we can help.


Kevin Dorrell
Luxembourg

[mmistrettammistretta]

Kevin,

Thanks for this response, it is a huge help.

So the bigger problem is this, we are using an F5 device, which in its simplest form, load balances traffic across server farms, and/or proxies connections to the servers. One function of the F5 is to use a virual IP address to access a server behind it.
The topology consists of 2 F5 devices, 2 6500 switches, fully meshed
The F5 devices work similarly to HSRP, using a virtual MAC and IP
They support 802.1q trunks. 2 VLANs are defined, 146 in front, and 111 behind the F5.
In the switches, there is an interface VLAN 146 to route traffic to/from the F5 to the devices in front of it.
VLAN 111 is used for the server farm behind it and is switched through the 6500's, no routing occurs. Only the F5 routes the traffic.
The F5 use STP passthrough, essentially allowing the BPDUs to pass through the interfaces, allowing the BPDU from switch1 to reach switch2.

Now onto the problem:

TCP traffic to the servers seems to work fine, although there have been some complaints of traffic problem.
ICMP fails 75% of the time to devices on the 111 network. Devices that ping, for monitoring, the 111 network, are sitting on the .5 network. We have confirmed routing is not an isssue. Any other device on the .5 network pings just fine to the 111.
The devices sending the echo requests are Nagios monitoring devices running on linux.
We have confirmed the request makes it to the host server, the reply is sent from the server, back through the F5, into the distribution switch1, and then just stops, seemingly discarded.
The behavior that we have seen is the request going to the F5 from the switch, is using source MAC of the interface, which you have confirmed it should be, but the return for some reason is using a different MAC, even though the ARP tables of the F5 have the correct MAC, so why is the MAC changing???
Additionally, the interface MAC of the active HSRP switch ends in c800, the standy is 8400. The HSRP MAC is ac05
So what we are seeing is the F5 sending echo replies to 840a, not a typo. Where is this coming from. All traffic to the F5 is using source of 8400.
Oddly, 8400 is the MAC of switch1, but the traffic is coming from switch2, why is switch2 using the MAC of switch1? Is this an HSRP thing? switch2 is the active.
when replies do succeed, they go to 8400, and interestingly, the seconf F5 still send to 840a, however, the responses go to switch1 which is STP blocking, so they go nowhere.

So the questions remaining are:

Where is the 840a coming from?
And why are the packets using the source MAC of a different switch thats not the active?


Thanks,
Mike

[Kevin Dorrell]

OK, I have thunk about it a bit, and I have a few questions and observations. I hope I have understood the architecture correctly, otherwise what I am about to ask will be garbage.

As I understand it, you have three VLANs: VLAN 5 which has the Nagios on it, VLAN 146 which faces the F5 pair, and LAN 111 which connects your F5s to your server farm. VLAN 111 is unrouted by the switches; it is just a layer-2 between the F5s and your server farm. VLAN 5 and VLAN 146 are routed by the switches, i.e. you have an SVI on each. VLAN 5 has an HSRP group 5 on it.

Do you have HSRP on the VLAN 146 interfaces of the switches, i.e. facing the F5s. What do the F5s have as their route back to the Nagios subnet. I would expect you to have HSRP between the two VLAN 146 SVIs, but in that case I would expect the F5s to route back to the virtual IP of the VLAN 146 HSRP group, and for there to be an ARP cache entry in the F5s ending in ac90 for that gateway address, and for the responses to be addressed to that virtual MAC. But if the two F5s are routing differently, and using interface MACs, then something is amiss.

So there seems to be something I have not quite grasped yet. Perhaps it would be useful to see the configs of your Vlan interfaces (5 and 146) on your switches, and something about your IP addressing scheme. (Change the prefixes if you want to disguise it.)

Oh, and what default route is the Nagios using? I would hope it is the virtual IP of HSRP group 5.


Kevin Dorrell
Luxembourg

[mmistrettammistretta]

Hi Kevin,
You are on the right track.

VLAN 146 is using HSRP

Switch1

interface Vlan146
description Virtual Connection to VLAN146
ip address x.x.146.3 255.255.255.0
no ip redirects
no ip proxy-arp
ip flow ingress
logging event link-status
standby 11 ip x.x.146.1
standby 11 timers 1 3
end

switch2

interface Vlan146
description Virtual Connection to VLAN146
ip address x.x.146.4 255.255.255.0
no ip redirects
no ip proxy-arp
ip flow ingress
logging event link-status
standby 11 ip x.x.146.1
standby 11 timers 1 3

standby 11 priority 110
end

Switch1

interface Vlan5
description Virtual Connection to VLAN5
ip address x.x.5.3 255.255.255.0
no ip redirects
no ip proxy-arp
ip flow ingress
logging event link-status
standby 5 ip x.x.5.1
standby 5 timers 1 3
standby 5 priority 110
standby 5 preempt delay minimum 1800
end

switch2

interface Vlan5
description Virtual Connection to VLAN5
ip address x.x.5.3 255.255.255.0
no ip redirects
no ip proxy-arp
ip flow ingress
logging event link-status
standby 5 ip x.x.5.1
standby 5 timers 1 3
standby 5 preempt delay minimum 1800
end


F5 is not using a route back to the server farm, the servers are on the same local subnet as the F5.
The servers on 111 or using 111.1, the "HSRP" address of the F5 as their default gateway.
The F5 is using the switch's HSRP address of 146.1 for its default route.
Traffic to the 111 subnet is using a static route in the distribution switches pointing to 146.5, the "HSRP" address of the F5.
(So traffic from Nagios, on .5, is getting to the F5 and the server farm, via a static route in the switch pointing to the F5.)
Nagios is using the default GW of 5.1 (HSRP)

Here is the arp cache of the F5

ARP x.x.146.1 - 00:00:0C:07:AC:0B VLAN VLAN_146 expire 277s resolved
ARP x.x.146.3 - 00:D0:05:2E:84:00 VLAN VLAN_146 expire 283s resolved
ARP x.x.146.4 - 00:D0:02:E5:C8:00 VLAN VLAN_146 expire 289s resolved

[Kevin Dorrell]

Just one minor thought: maybe you should have preempt on vlan 146 as well as on vlan 5. Otherwise you might end up with asymmetric routing following any network disruption. You can check this by seeing which switch is HSRP active for each vlan.

Another question: which switch is the root of vlan 146? I presume it would be the one that is HSRP active, but I'm not sure.

You say the switches and the F5s are a full mesh, so I presume the two switches are tied together other than through the F5s. I would also presume that the Spanning-Tree cost of that tie bar is less than that of the links to the F5s. I imagine that the two switches are tied together, and that each F5 has two trunks, one to each switch, and each trunk carrying 111 and 146. Right?

I am a bit worried about what you said about the F5s doing a SpanningTree passthrough. That is the bit I still have to think about. That would put STP blocking on the F5-facing ports of the standby switch, which would make it vitally important that the spanning tree root be the same as the HSRP active.

On the evidence so far, I would expect the F5s to be delivering their responses to 00:00:0C:07:AC:0B.


Kevin Dorrell
Luxembourg

[mmistrettammistretta]

Hi Kevin,

I am not sure how i missed that line, but preempt is configured on 146 also.
The STP roots do match the HSRP active as well.
We did find it is assymetric and will be updating this, as well, we are looking to move to HSRP v2 so we can match the group to the VLAN number.
As for the STP passthrough, my understanding is that BPDUs received by the F5, are passed from one port to the next, and to the other swith, allowing the other switch to essentially see the other switch as its STP "neighbor"
STP blocking is on switch1 facing the F5, for 111 and 146, which are the only allowed VLANs to the F5.
I also would have expected the HSRP MAC to be the destination, which sparked my orignal question of the thread
And not only is it the interface MAC, but the MAC of the standby (146).
With this being said, i make an additional observation.
Traffic from Nagios uses switch1 as its default gateway (HSRP active), thus the source MAC to F5 will be this switch. It is switched through switch2 and forwarded to the F5. The F5 however should be using switch2 as the Default GW, its HSRP active. (It is assymetric as you mention)
Somehow the F5 has obtained the IP addresses of the 2 switches, not sure how. It does use ICMP to monitor the swithces (not sure why). It has mapped their IP to their physical address, again not sure how, but it seems to have populated its ARP cache based on this (but why would it populate ARP based on ICMP reply, unless its not following standards). It does also have the HSRP IP and MAC mapped correctly.
So it looks like its using layer 2 logic for Layer 3 forwarding, and mapping the layer 2 address to the interface received on, rather than using the ARP table.
But then there's still the mystery of the incorrect MAC

[Kevin Dorrell]

Mike,

If we suspect the F5s are somehow populating their ARP cache from the data traffic, it might be best to abandon the asymmetric routing. That is, make the HSRP the same from both VLANs. Just use one switch as the active and the other as a standby, on both VLANs. At the moment you have switch 1 active on VLAN 5, but switch 2 active on VLAN 146. I don't really see any advantage in doing this.


Kevin Dorrell
Luxembourg