Networks that ignore ICMPv6 PTBs: Difference between revisions
No edit summary |
No edit summary |
||
| (27 intermediate revisions by the same user not shown) | |||
| Line 1: | Line 1: | ||
In [[IPv6]] networks, an ICMPv6 packet too big (PTB) message is sent by an intermediary router when a destination's outgoing interface has an MTU lower than the packet being routed toward it. In the IPv4 world, the router might fragment this packet but in the IPv6 world this is forbidden (as it should be). This signals the sending host to either lower the TCP MSS (in the case of TCP) and retry with the lower MTU or return an error to the application in the case of most other protocols, like UDP. This is called path MTU discovery (PMTUD) and a similar process takes place in [[IPv4]] networks. We normally see ICMPv6 PTBs emitted when a router is connected to both Ethernet (e.g. 1500 bytes MTU) and a tunnel interface (e.g. 1280 bytes MTU). VPNs and IPv6-over-IPv4 tunnels make use of ICMPv6 PTBs. | |||
If everything works as expected, the end user's application will work properly without a hitch. | If everything works as expected, the end user's application will work properly without a hitch. | ||
| Line 10: | Line 10: | ||
Connecting to my.t-mobile.com (my.t-mobile.com)|2a02:e980:14::90|:443... connected.</pre> | Connecting to my.t-mobile.com (my.t-mobile.com)|2a02:e980:14::90|:443... connected.</pre> | ||
TCP connects fine since the 3-way handshake consists of small packets, but the TLS handshake fails leaving wget just sitting there until it hits some sort of application-level timeout. Here's what it might look like on the network: | TCP connects fine since the 3-way handshake consists of small packets, but the TLS handshake fails leaving wget just sitting there until it hits some sort of application-level timeout. Here's what it might look like on the network (we are only seeing one direction of the TCP connection at the moment due to routing policy): | ||
<pre>(trident:17:59:PST)% sudo tcpdump -i any -ns0 host 2a02:e980:14::90 | <pre>(trident:17:59:PST)% sudo tcpdump -i any -ns0 host 2a02:e980:14::90 | ||
| Line 39: | Line 39: | ||
[...snip...]</pre> | [...snip...]</pre> | ||
As seen above, excalibur's interface with address 2001:470:d6:61::2 (the local end of a Hurricane Electric tunnel) is receiving packets with size 1388 and can't shove it into the outgoing interface that is set to an MTU of 1280. It keeps sending the ICMPv6 PTB messages to 2a02:e980:14::90 but the packets still keep coming with the large size. | |||
[https://tools.ietf.org/html/rfc6555 Happy Eyeballs] does not account for this since the connection ''is'' established and the application (web browser, for example) has no way of knowing if the remote host is just very busy or the return traffic is actually getting dropped. | [https://tools.ietf.org/html/rfc6555 Happy Eyeballs] does not account for this since the connection ''is'' established and the application (web browser, for example) has no way of knowing if the remote host is just very busy or the return traffic is actually getting dropped. | ||
Note that large networks run by Amazon and Google haven't even gotten this 100% correct in the past, so apparently it's easily broken. See [https://twitter.com/jrmassar/status/586646152923246594 here] and [https://twitter.com/Prolixium/status/557018191919337474 here]. | |||
= The List = | = The List = | ||
Here's a list of known sites that | Here's a list of known sites that have been known to not load correctly due to their hosting provider ignoring ICMPv6 PTBs. | ||
{|border="1" cellpadding="2" | {|border="1" cellpadding="2" | ||
| Line 54: | Line 58: | ||
!Notes | !Notes | ||
|- | |- | ||
|[https://my.t-mobile.com/ my.t-mobile.com] || [https://www.incapsula.com/ Incapsula] || 2a02:e980:14::90 || [http://bgp.he.net/AS19551 AS19551] || | |[https://my.t-mobile.com/ my.t-mobile.com] || [https://www.incapsula.com/ Incapsula] || 2a02:e980:14::90 || [http://bgp.he.net/AS19551 AS19551] || 2019-02-01 || Ironic. | ||
|- | |||
|[https://dcp2.att.com/OEPNDClient/ dcp2.att.com/OEPNDClient/] || [http://att.com/ AT&T] || 2001:1890:1c00:2401::4:1008 || [http://bgp.he.net/AS7018 AS7018] || 2019-02-01 || This website is used to manage Apple iPad cellular plans. | |||
|- | |- | ||
|[ | |[http://www.northbrunswicknj.gov/ www.northbrunswicknj.gov] || [https://www.cloudaccess.net/ CloudAccess.net] || 2607:1b00:93b2:e42c::19c4 || [https://bgp.he.net/AS54456 AS54456] || 2019-02-01 || Tested from New Jersey, USA | ||
|} | |} | ||
The above list may be skewed toward destinations that are on the west coast of the USA due to GeoIP, since that is where most of the tests originate. FYI, shout out to [http://www.mythic-beasts.com/ Mythic Beasts Ltd] ([https://bgp.he.net/AS44684 AS44864]) for being very responsive when I erroneously pointed the finger at them, recently. It would be nice if all network operators could be responsive even if the group reporting the problem isn't a direct customer. | |||
Latest revision as of 02:31, 2 February 2019
In IPv6 networks, an ICMPv6 packet too big (PTB) message is sent by an intermediary router when a destination's outgoing interface has an MTU lower than the packet being routed toward it. In the IPv4 world, the router might fragment this packet but in the IPv6 world this is forbidden (as it should be). This signals the sending host to either lower the TCP MSS (in the case of TCP) and retry with the lower MTU or return an error to the application in the case of most other protocols, like UDP. This is called path MTU discovery (PMTUD) and a similar process takes place in IPv4 networks. We normally see ICMPv6 PTBs emitted when a router is connected to both Ethernet (e.g. 1500 bytes MTU) and a tunnel interface (e.g. 1280 bytes MTU). VPNs and IPv6-over-IPv4 tunnels make use of ICMPv6 PTBs.
If everything works as expected, the end user's application will work properly without a hitch.
If the ICMPv6 PTB is blocked or ignored by the sending host or network, then connections will hang inexplicably. For example:
(destiny:17:43:PST)% wget https://my.t-mobile.com/ --2017-12-30 17:56:46-- https://my.t-mobile.com/ Resolving my.t-mobile.com (my.t-mobile.com)... 2a02:e980:14::90, 192.230.67.144 Connecting to my.t-mobile.com (my.t-mobile.com)|2a02:e980:14::90|:443... connected.
TCP connects fine since the 3-way handshake consists of small packets, but the TLS handshake fails leaving wget just sitting there until it hits some sort of application-level timeout. Here's what it might look like on the network (we are only seeing one direction of the TCP connection at the moment due to routing policy):
(trident:17:59:PST)% sudo tcpdump -i any -ns0 host 2a02:e980:14::90 tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on any, link-type LINUX_SLL (Linux cooked), capture size 262144 bytes 18:00:04.918923 IP6 2a02:e980:14::90.443 > 2620:6:2000:105:21c:c0ff:feb2:8dbd.53684: Flags [S.], seq 994050440, ack 2921061295, win 27760, options [mss 1400,sackOK,TS val 2972444571 ecr 822082903,nop,wscale 7], length 0 18:00:04.918985 IP6 2a02:e980:14::90.443 > 2620:6:2000:105:21c:c0ff:feb2:8dbd.53684: Flags [S.], seq 994050440, ack 2921061295, win 27760, options [mss 1400,sackOK,TS val 2972444571 ecr 822082903,nop,wscale 7], length 0 18:00:05.059249 IP6 2a02:e980:14::90.443 > 2620:6:2000:105:21c:c0ff:feb2:8dbd.53684: Flags [.], ack 255, win 226, options [nop,nop,TS val 2972444606 ecr 822082937], length 0 18:00:05.059267 IP6 2a02:e980:14::90.443 > 2620:6:2000:105:21c:c0ff:feb2:8dbd.53684: Flags [.], ack 255, win 226, options [nop,nop,TS val 2972444606 ecr 822082937], length 0 18:00:05.059324 IP6 2a02:e980:14::90.443 > 2620:6:2000:105:21c:c0ff:feb2:8dbd.53684: Flags [.], seq 1:1389, ack 255, win 226, options [nop,nop,TS val 2972444606 ecr 822082937], length 1388 18:00:05.059608 IP6 2001:470:d6:61::2 > 2a02:e980:14::90: ICMP6, packet too big, mtu 1436, length 1240 18:00:05.059663 IP6 2a02:e980:14::90.443 > 2620:6:2000:105:21c:c0ff:feb2:8dbd.53684: Flags [P.], seq 1389:4097, ack 255, win 226, options [nop,nop,TS val 2972444606 ecr 822082937], length 2708 18:00:05.059678 IP6 2a02:e980:14::90.443 > 2620:6:2000:105:21c:c0ff:feb2:8dbd.53684: Flags [P.], seq 1389:4097, ack 255, win 226, options [nop,nop,TS val 2972444606 ecr 822082937], length 2708 18:00:05.060659 IP6 2a02:e980:14::90.443 > 2620:6:2000:105:21c:c0ff:feb2:8dbd.53684: Flags [.], seq 4097:5485, ack 255, win 226, options [nop,nop,TS val 2972444606 ecr 822082937], length 1388 18:00:05.060725 IP6 2001:470:d6:61::2 > 2a02:e980:14::90: ICMP6, packet too big, mtu 1436, length 1240 18:00:05.333717 IP6 2a02:e980:14::90.443 > 2620:6:2000:105:21c:c0ff:feb2:8dbd.53684: Flags [P.], seq 6873:7717, ack 255, win 226, options [nop,nop,TS val 2972444675 ecr 822082937], length 844 18:00:05.333766 IP6 2a02:e980:14::90.443 > 2620:6:2000:105:21c:c0ff:feb2:8dbd.53684: Flags [P.], seq 6873:7717, ack 255, win 226, options [nop,nop,TS val 2972444675 ecr 822082937], length 844 18:00:05.476362 IP6 2a02:e980:14::90.443 > 2620:6:2000:105:21c:c0ff:feb2:8dbd.53684: Flags [.], seq 1:1389, ack 255, win 226, options [nop,nop,TS val 2972444710 ecr 822083041], length 1388 18:00:05.476492 IP6 2001:470:d6:61::2 > 2a02:e980:14::90: ICMP6, packet too big, mtu 1436, length 1240 18:00:05.881759 IP6 2a02:e980:14::90.443 > 2620:6:2000:105:21c:c0ff:feb2:8dbd.53684: Flags [.], seq 1:1389, ack 255, win 226, options [nop,nop,TS val 2972444812 ecr 822083041], length 1388 18:00:05.881960 IP6 2001:470:d6:61::2 > 2a02:e980:14::90: ICMP6, packet too big, mtu 1436, length 1240 18:00:06.697584 IP6 2a02:e980:14::90.443 > 2620:6:2000:105:21c:c0ff:feb2:8dbd.53684: Flags [.], seq 1:1389, ack 255, win 226, options [nop,nop,TS val 2972445016 ecr 822083041], length 1388 18:00:06.697775 IP6 2001:470:d6:61::2 > 2a02:e980:14::90: ICMP6, packet too big, mtu 1436, length 1240 18:00:08.334780 IP6 2a02:e980:14::90.443 > 2620:6:2000:105:21c:c0ff:feb2:8dbd.53684: Flags [.], seq 1:1389, ack 255, win 226, options [nop,nop,TS val 2972445425 ecr 822083041], length 1388 18:00:08.334933 IP6 2001:470:d6:61::2 > 2a02:e980:14::90: ICMP6, packet too big, mtu 1436, length 1240 18:00:11.610214 IP6 2a02:e980:14::90.443 > 2620:6:2000:105:21c:c0ff:feb2:8dbd.53684: Flags [.], seq 1:1389, ack 255, win 226, options [nop,nop,TS val 2972446244 ecr 822083041], length 1388 18:00:11.610914 IP6 2001:470:d6:61::2 > 2a02:e980:14::90: ICMP6, packet too big, mtu 1436, length 1240 [...snip...]
As seen above, excalibur's interface with address 2001:470:d6:61::2 (the local end of a Hurricane Electric tunnel) is receiving packets with size 1388 and can't shove it into the outgoing interface that is set to an MTU of 1280. It keeps sending the ICMPv6 PTB messages to 2a02:e980:14::90 but the packets still keep coming with the large size.
Happy Eyeballs does not account for this since the connection is established and the application (web browser, for example) has no way of knowing if the remote host is just very busy or the return traffic is actually getting dropped.
Note that large networks run by Amazon and Google haven't even gotten this 100% correct in the past, so apparently it's easily broken. See here and here.
The List
Here's a list of known sites that have been known to not load correctly due to their hosting provider ignoring ICMPv6 PTBs.
| Site | Website Host or CDN | Host or CDN's ASN | Website IP Address Observed | Last Tested | Notes |
|---|---|---|---|---|---|
| my.t-mobile.com | Incapsula | 2a02:e980:14::90 | AS19551 | 2019-02-01 | Ironic. |
| dcp2.att.com/OEPNDClient/ | AT&T | 2001:1890:1c00:2401::4:1008 | AS7018 | 2019-02-01 | This website is used to manage Apple iPad cellular plans. |
| www.northbrunswicknj.gov | CloudAccess.net | 2607:1b00:93b2:e42c::19c4 | AS54456 | 2019-02-01 | Tested from New Jersey, USA |
The above list may be skewed toward destinations that are on the west coast of the USA due to GeoIP, since that is where most of the tests originate. FYI, shout out to Mythic Beasts Ltd (AS44864) for being very responsive when I erroneously pointed the finger at them, recently. It would be nice if all network operators could be responsive even if the group reporting the problem isn't a direct customer.