NetworkNick.net

So I was at a client site this week troubleshooting a strange traffic mirroring issue. It went something like this:
I thought the ProCurve 3500 mirror port was dropping specific inbound oversized SIP packets on the outbound mirror port. The actual traffic went through, but the mirror port only saw the continuation frame, not the initial 1500 bytes frame. This was a standard 1 port>1port mirror configuration, bi-directional, no filtering. So we dug in to find out why the mirror port wasn’t forwarding that first frame. For a while we were stumped, but then I did a netstat -e on the sniffer (Windows based) and whatdya know, it’s got 63,000 unknown frames…! So here is what happend:

When you mirror an interface on the ProCurve switch, even locally, any traffic sent outbound on the monitored port has a VLAN tag appended for the VLAN it is part of, even when you are monitoring an untagged port. When you are sniffing traffic with a non 802.1q compliant NIC, and the packet size exceeds 1514, (you get fragmentation obviously). HOWEVER, because the total packet size is 1514+4 bytes for the VLAN tag outbound, the sniffer NIC drops it completely as it is oversized. This was causing the sniffer to miss the first packet (total size~1800 bytes) and just see the second fragment(~300 bytes). Once we connected a 802.1q compliant NIC to the system, we saw all packets.Moral of the story: Use a .1q compliant NIC if you what to see all bidirectional traffic on a ProCurve mirror port. If you don’t have a .1q compliant NIC, make sure to enable the .1p setting in the driver, this will allow it to recieve the oversized packet (although it won’t interperet the VLAN tag).

I speaking with one of the developers, the intent of appending the tag was to allow an IDS system to delineate traffic from multiple networks on a single upling/mirror session, but this obviously breaks some expected behaviors elsewhere. ProCurve is going to be releasing an enhancement that allows you to disable the VLAN tag on the mirror session, but until them, make sure your NIC can take the tag!