MTU issue routing traffic via Cisco GRE tunnel to Nokia/Check Point firewall

[m***@convergys.com]

We have been suffering an issue to do with Checkpoint, Cisco GRE tunnels
and MTU size for a number of months now, and I thought it might be worth
posting a description of our problem on this list in case someone is able
to help. We feel that we have exhausted most avenues of trying to
troubleshoot this issue.

What we are trying to do is route Internet traffic for remote branch office
sites via our central office's Internet connection. As an example, we have
a 2Mb AT&T Internet connection in our Paris office, connected to a 15Mb
AT&T Internet connection in London. We run a Cisco GRE tunnel between a
3640-VPN/MP router in Paris and a 7206VXR/G1 router in London. In London,
we also have a Nokia IP530 appliance running a fresh install of Check Point
NG:AI, connected to a 10Mb PSINet Internet connection.

The Cisco GRE tunnel has a MTU size of 1420 set at both ends for it's
tunnel interfaces. This is the highest we can use based on the
encryption/encapsulation chosen in order to facilitate protocols such as
OSPF from working over the link. All other interfaces along the way
(router ethernets and Nokia interfaces) are set the default 1500.

The problem is that users in the Paris branch office are unable to view
_some_ websites. Examples of ones that don't work are www.yahoo.fr and
www.adp.fr. The majority work fine, including www.cisco.com and
www.google.com.

When running a tcpdump on the IP530 in London (on the external interface),
during a session from Paris to one of the offending websites, the following
is logged:

16:36:21.025051 O 154.38.47.5 > 194.3.182.10: icmp: 154.38.47.5 unreachable
- need to frag (mtu 1420)
16:36:27.586541 I 194.3.182.10.80 > 154.38.47.5.41571: . 1:1461(1460) ack
249 win 63992 (DF)
16:36:27.588356 O 154.38.47.5 > 194.3.182.10: icmp: 154.38.47.5 unreachable
- need to frag (mtu 1420)
16:36:40.711277 I 194.3.182.10.80 > 154.38.47.5.41571: . 1:1461(1460) ack
249 win 63992 (DF)
16:36:40.713043 O 154.38.47.5 > 194.3.182.10: icmp: 154.38.47.5 unreachable
- need to frag (mtu 1420)

We have also noticed that the packet size of traffic received from
offending sites seems to be 1514 bytes. For sites that work, i.e.
cisco.com, it seems to be 1486 bytes.

We have tried lots of things on the GRE tunnel configuration on our Cisco
routers, including settings to ignore the Don't Fragment (DF) bit, and to
force different MTU sizes. A long-running Cisco TAC case has not suggested
any way around our problem.

Can anyone explain the cause of this problem, and suggest anything that can
be done on our Nokia/Check Point configuration to prevent this occurring?
Out of interest, when we route the Internet traffic past the Nokia IP530
firewall and onto an Internet connection at another downstream site, which
uses a Cisco PIX firewall instead, the remote Paris users ARE able to
browse the offending websites successfully. This indicates that it must be
something to do with the Nokia/Check Point installation.

Any comments or suggestions would be greatly received.

Thanks,
Marcel

--
"NOTICE: The information contained in this electronic mail transmission is
intended by Convergys Corporation for the use of the named individual or
entity to which it is directed and may contain information that is
privileged or otherwise confidential. If you have received this electronic
mail transmission in error, please delete it from your system without
copying or forwarding it, and notify the sender of the error by reply email
or by telephone (collect), so that the sender's address records can be
corrected."

[r***@basf-it-services.com]

Post by m***@convergys.com
We have been suffering an issue to do with Checkpoint, Cisco GRE tunnels
and MTU size for a number of months now, and I thought it might be worth
posting a description of our problem on this list in case someone is able
to help. We feel that we have exhausted most avenues of trying to
troubleshoot this issue.

You might try the solution SK14995 of Check Point's knowledge base. It
suggest to set "fw_clamp_tcp_mss = true" in objects_5_0.C in conjunction
with tuning the MTU size of the Interface in question.

Best regards,
Rainer

[Ben Nagy]

Ok, I have a working theory. Stop me if you've heard this one... ;)

It's PMTU-D. Again.

Just confirm that someone hasn't helpfully "tightened" your firewall
settings to deny all outbound ICMP errors. Over-enthusiastic firewall
monkeys seem to do that fairly often.

If those ICMP unreachables aren't actually getting back through the firewall
to the sending host (the outside webserver) then it will be breaking path
MTU discovery, and you'll get symptoms like what you're seeing.

As a workaround, you can lower the MTU on your Paris LAN hosts. This will
make sure that the client never asks for an MSS big enough to cause the
problem. I guess 1380 would be the magic number there, but I haven't
actually checked the overheads. That's a horribly ugly thing to do, by the
way, and I feel kind of bad for suggesting it.

'luck...

ben

(Oh, and let me know the result? I like mysteries.)

[1] http://www.ietf.org/rfc/rfc1191.txt

-----Original Message-----
Sent: Thursday, December 04, 2003 12:24 PM
Subject: [fw-wiz] MTU issue routing traffic via Cisco GRE
tunnel to Nokia/Check Point firewall
We have been suffering an issue to do with Checkpoint, Cisco
GRE tunnels and MTU size for a number of months now[...]

[...]

The Cisco GRE tunnel has a MTU size of 1420 set at both ends
for it's tunnel interfaces. This is the highest we can use
based on the encryption/encapsulation chosen in order to
facilitate protocols such as OSPF from working over the link.
All other interfaces along the way (router ethernets and
Nokia interfaces) are set the default 1500.

[...]

When running a tcpdump on the IP530 in London (on the
external interface), during a session from Paris to one of

[...]

16:36:27.586541 I 194.3.182.10.80 > 154.38.47.5.41571: .
1:1461(1460) ack
249 win 63992 (DF)
154.38.47.5 unreachable
- need to frag (mtu 1420)

[...]

Out of interest, when we route the Internet traffic past the
Nokia IP530 firewall and onto an Internet connection at
another downstream site, which uses a Cisco PIX firewall
instead, the remote Paris users ARE able to browse the
offending websites successfully. This indicates that it must
be something to do with the Nokia/Check Point installation.

[edp]

Post by m***@convergys.com
We have tried lots of things on the GRE tunnel configuration on our Cisco
routers, including settings to ignore the Don't Fragment (DF) bit, and to
force different MTU sizes. A long-running Cisco TAC case has not suggested
any way around our problem.

Seems also to me a path mtu discovery problem.
Maybe non-working webservers send packets bigger than your gre tunnel
mtu and - more important - with DF set in ip headers; when this packets
is processed by your router interface, your router cannot fragment the
packet keeping forwarding going on, because it honors the DF flag and so
it generates a icmp "require fragmentation" to the webserver in order to
force the webserver to produce smaller packets. But maybe this icmp got
lost in transit due to strict filters so the communication stalls.

Investigate your appliance feature, maybe you can patch in-transit
client TCP MSS in order to avoid fragmentation.

Regards,
.FT

[Eric Vyncke]

Late reply... sorry about it.

It looks like you are experimenting a Path MTU Discovery issue. As it is quite possible that firewalls (or even a load balancers somewhere -- like in Yahoo) simply drop the required ICMP messages.

The simple solutions are:
- - use 'ip tcp adjust-mss 1400' on a router seeing traffic in the clear to force MSS to 1400 so IP datagram size to 1420 (of course 1400 is just a guess), this will cover all TCP traffic
- - set 'ip mtu 1500' on the GRE tunnel interface (yes 1500 bytes)

The latter will also work for non TCP traffic. So, you probably need both.

Of course, the 'clean' solution is to let the ICMP unreachable too big messages go through.

Reasoning:
- - GRE encapsulation clears the DF bit UNLESS 'tunnel path-mtu-discovery' is set on the tunnel interface (if turned on the tunnel MTU will be dynamically adjusted upon receipt of ICMP)
- - IPsec encapsulation copies the DF and adjusts the path MTU upon receipt of ICMP UNLESS 'crypto ipsec df-bit clear/set' is configured in the crypto map
- - router will fragment when forwarding to any interface whose MTU is smaller than the received IP packet. This happens often when forwarding to a GRE tunnel whose MTU is 1476 per default...

The last point forces the router to drop all 1500-bytes packets and to send an ICMP message when a DF packet is received.

Hope it helps

- -eric

Post by m***@convergys.com
We have been suffering an issue to do with Checkpoint, Cisco GRE tunnels
and MTU size for a number of months now, and I thought it might be worth
posting a description of our problem on this list in case someone is able
to help. We feel that we have exhausted most avenues of trying to
troubleshoot this issue.
What we are trying to do is route Internet traffic for remote branch office
sites via our central office's Internet connection. As an example, we have
a 2Mb AT&T Internet connection in our Paris office, connected to a 15Mb
AT&T Internet connection in London. We run a Cisco GRE tunnel between a
3640-VPN/MP router in Paris and a 7206VXR/G1 router in London. In London,
we also have a Nokia IP530 appliance running a fresh install of Check Point
NG:AI, connected to a 10Mb PSINet Internet connection.
The Cisco GRE tunnel has a MTU size of 1420 set at both ends for it's
tunnel interfaces. This is the highest we can use based on the
encryption/encapsulation chosen in order to facilitate protocols such as
OSPF from working over the link. All other interfaces along the way
(router ethernets and Nokia interfaces) are set the default 1500.
The problem is that users in the Paris branch office are unable to view
_some_ websites. Examples of ones that don't work are www.yahoo.fr and
www.adp.fr. The majority work fine, including www.cisco.com and
www.google.com.
When running a tcpdump on the IP530 in London (on the external interface),
during a session from Paris to one of the offending websites, the following
16:36:21.025051 O 154.38.47.5 > 194.3.182.10: icmp: 154.38.47.5 unreachable
- need to frag (mtu 1420)
16:36:27.586541 I 194.3.182.10.80 > 154.38.47.5.41571: . 1:1461(1460) ack
249 win 63992 (DF)
16:36:27.588356 O 154.38.47.5 > 194.3.182.10: icmp: 154.38.47.5 unreachable
- need to frag (mtu 1420)
16:36:40.711277 I 194.3.182.10.80 > 154.38.47.5.41571: . 1:1461(1460) ack
249 win 63992 (DF)
16:36:40.713043 O 154.38.47.5 > 194.3.182.10: icmp: 154.38.47.5 unreachable
- need to frag (mtu 1420)
We have also noticed that the packet size of traffic received from
offending sites seems to be 1514 bytes. For sites that work, i.e.
cisco.com, it seems to be 1486 bytes.
We have tried lots of things on the GRE tunnel configuration on our Cisco
routers, including settings to ignore the Don't Fragment (DF) bit, and to
force different MTU sizes. A long-running Cisco TAC case has not suggested
any way around our problem.
Can anyone explain the cause of this problem, and suggest anything that can
be done on our Nokia/Check Point configuration to prevent this occurring?
Out of interest, when we route the Internet traffic past the Nokia IP530
firewall and onto an Internet connection at another downstream site, which
uses a Cisco PIX firewall instead, the remote Paris users ARE able to
browse the offending websites successfully. This indicates that it must be
something to do with the Nokia/Check Point installation.
Any comments or suggestions would be greatly received.
Thanks,
Marcel
--
"NOTICE: The information contained in this electronic mail transmission is
intended by Convergys Corporation for the use of the named individual or
entity to which it is directed and may contain information that is
privileged or otherwise confidential. If you have received this electronic
mail transmission in error, please delete it from your system without
copying or forwarding it, and notify the sender of the error by reply email
or by telephone (collect), so that the sender's address records can be
corrected."
_______________________________________________
firewall-wizards mailing list
http://honor.icsalabs.com/mailman/listinfo/firewall-wizards

[edp]

Post by Eric Vyncke
- - use 'ip tcp adjust-mss 1400' on a router seeing traffic in the

clear to > force MSS to 1400 so IP datagram size to 1420 (of course 1400
is just a >guess), this will cover all TCP traffic

Post by Eric Vyncke
- - set 'ip mtu 1500' on the GRE tunnel interface (yes 1500 bytes)

Just for clarity, with a mss option set to 1400, the ip packet size will
be 1440.