Networks that ignore ICMPv6 PTBs

In IPv6 networks, an ICMPv6 packet too big (PTB) message is sent by an intermediary router when a destination's outgoing interface has an MTU lower than the packet being routed toward it. In the IPv4 world, the router might fragment this packet but in the IPv6 world this is forbidden (as it should be). This signals the sending host to either lower the TCP MSS (in the case of TCP) and retry with the lower MTU or return an error to the application in the case of most other protocols, like UDP. We normally see ICMPv6 PTBs emitted when a router is connected to both Ethernet (e.g. 1500 bytes MTU) and a tunnel interface (e.g. 1280 bytes MTU). VPNs and IPv6-over-IPv4 tunnels make use of ICMPv6 PTBs.

If everything works as expected, the end user's application will work properly without a hitch.

If the ICMPv6 PTB is blocked or ignored by the sending host or network, then connections will hang inexplicably. For example:

(destiny:17:43:PST)% wget https://my.t-mobile.com/
--2017-12-30 17:56:46--  https://my.t-mobile.com/
Resolving my.t-mobile.com (my.t-mobile.com)... 2a02:e980:14::90, 192.230.67.144
Connecting to my.t-mobile.com (my.t-mobile.com)|2a02:e980:14::90|:443... connected.

TCP connects fine since the 3-way handshake consists of small packets, but the TLS handshake fails leaving wget just sitting there until it hits some sort of application-level timeout. Here's what it might look like on the network (we are only seeing one direction of the TCP connection at the moment due to routing policy):

(trident:17:59:PST)% sudo tcpdump -i any -ns0 host 2a02:e980:14::90
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on any, link-type LINUX_SLL (Linux cooked), capture size 262144 bytes
18:00:04.918923 IP6 2a02:e980:14::90.443 > 2620:6:2000:105:21c:c0ff:feb2:8dbd.53684: Flags [S.], seq 994050440, ack 2921061295, win 27760, options [mss 1400,sackOK,TS val 2972444571 ecr 822082903,nop,wscale 7], length 0
18:00:04.918985 IP6 2a02:e980:14::90.443 > 2620:6:2000:105:21c:c0ff:feb2:8dbd.53684: Flags [S.], seq 994050440, ack 2921061295, win 27760, options [mss 1400,sackOK,TS val 2972444571 ecr 822082903,nop,wscale 7], length 0
18:00:05.059249 IP6 2a02:e980:14::90.443 > 2620:6:2000:105:21c:c0ff:feb2:8dbd.53684: Flags [.], ack 255, win 226, options [nop,nop,TS val 2972444606 ecr 822082937], length 0
18:00:05.059267 IP6 2a02:e980:14::90.443 > 2620:6:2000:105:21c:c0ff:feb2:8dbd.53684: Flags [.], ack 255, win 226, options [nop,nop,TS val 2972444606 ecr 822082937], length 0
18:00:05.059324 IP6 2a02:e980:14::90.443 > 2620:6:2000:105:21c:c0ff:feb2:8dbd.53684: Flags [.], seq 1:1389, ack 255, win 226, options [nop,nop,TS val 2972444606 ecr 822082937], length 1388
18:00:05.059608 IP6 2001:470:d6:61::2 > 2a02:e980:14::90: ICMP6, packet too big, mtu 1436, length 1240
18:00:05.059663 IP6 2a02:e980:14::90.443 > 2620:6:2000:105:21c:c0ff:feb2:8dbd.53684: Flags [P.], seq 1389:4097, ack 255, win 226, options [nop,nop,TS val 2972444606 ecr 822082937], length 2708
18:00:05.059678 IP6 2a02:e980:14::90.443 > 2620:6:2000:105:21c:c0ff:feb2:8dbd.53684: Flags [P.], seq 1389:4097, ack 255, win 226, options [nop,nop,TS val 2972444606 ecr 822082937], length 2708
18:00:05.060659 IP6 2a02:e980:14::90.443 > 2620:6:2000:105:21c:c0ff:feb2:8dbd.53684: Flags [.], seq 4097:5485, ack 255, win 226, options [nop,nop,TS val 2972444606 ecr 822082937], length 1388
18:00:05.060725 IP6 2001:470:d6:61::2 > 2a02:e980:14::90: ICMP6, packet too big, mtu 1436, length 1240
18:00:05.333717 IP6 2a02:e980:14::90.443 > 2620:6:2000:105:21c:c0ff:feb2:8dbd.53684: Flags [P.], seq 6873:7717, ack 255, win 226, options [nop,nop,TS val 2972444675 ecr 822082937], length 844
18:00:05.333766 IP6 2a02:e980:14::90.443 > 2620:6:2000:105:21c:c0ff:feb2:8dbd.53684: Flags [P.], seq 6873:7717, ack 255, win 226, options [nop,nop,TS val 2972444675 ecr 822082937], length 844
18:00:05.476362 IP6 2a02:e980:14::90.443 > 2620:6:2000:105:21c:c0ff:feb2:8dbd.53684: Flags [.], seq 1:1389, ack 255, win 226, options [nop,nop,TS val 2972444710 ecr 822083041], length 1388
18:00:05.476492 IP6 2001:470:d6:61::2 > 2a02:e980:14::90: ICMP6, packet too big, mtu 1436, length 1240
18:00:05.881759 IP6 2a02:e980:14::90.443 > 2620:6:2000:105:21c:c0ff:feb2:8dbd.53684: Flags [.], seq 1:1389, ack 255, win 226, options [nop,nop,TS val 2972444812 ecr 822083041], length 1388
18:00:05.881960 IP6 2001:470:d6:61::2 > 2a02:e980:14::90: ICMP6, packet too big, mtu 1436, length 1240
18:00:06.697584 IP6 2a02:e980:14::90.443 > 2620:6:2000:105:21c:c0ff:feb2:8dbd.53684: Flags [.], seq 1:1389, ack 255, win 226, options [nop,nop,TS val 2972445016 ecr 822083041], length 1388
18:00:06.697775 IP6 2001:470:d6:61::2 > 2a02:e980:14::90: ICMP6, packet too big, mtu 1436, length 1240
18:00:08.334780 IP6 2a02:e980:14::90.443 > 2620:6:2000:105:21c:c0ff:feb2:8dbd.53684: Flags [.], seq 1:1389, ack 255, win 226, options [nop,nop,TS val 2972445425 ecr 822083041], length 1388
18:00:08.334933 IP6 2001:470:d6:61::2 > 2a02:e980:14::90: ICMP6, packet too big, mtu 1436, length 1240
18:00:11.610214 IP6 2a02:e980:14::90.443 > 2620:6:2000:105:21c:c0ff:feb2:8dbd.53684: Flags [.], seq 1:1389, ack 255, win 226, options [nop,nop,TS val 2972446244 ecr 822083041], length 1388
18:00:11.610914 IP6 2001:470:d6:61::2 > 2a02:e980:14::90: ICMP6, packet too big, mtu 1436, length 1240

[...snip...]

As seen above, excalibur's interface with address 2001:470:d6:61::2 (the local end of a Hurricane Electric tunnel) is receiving packets with size 1388 and can't shove it into the outgoing interface that is set to an MTU of 1280. It keeps sending the ICMPv6 PTB messages to 2a02:e980:14::90 but the packets still keep coming with the large size.

Happy Eyeballs does not account for this since the connection is established and the application (web browser, for example) has no way of knowing if the remote host is just very busy or the return traffic is actually getting dropped.

The List

Here's a list of known sites that appear to ignore ICMPv6 PTBs. Note that Google hasn't even gotten this 100% correct in the past, so apparently it's easily broken.

Site	Website Host or CDN	Host or CDN's ASN	Website IP Address Observed	Last Tested	Notes
my.t-mobile.com	Incapsula	2a02:e980:14::90	AS19551	2017-12-30	N/A
xkcd.com	Fastly	2a04:4e42:400::67 2a04:4e42:600::67 2a04:4e42::67 2a04:4e42:200::67	AS54113	2017-12-30	N/A
dcp2.att.com/OEPNDClient/	AT&T	2001:1890:1c00:2401::4:1008	AS7018	2017-12-31	This websute is used to manage Apple iPad cellular plans.