IPv6 Troubleshooting

Joni Julian

Ohio LinuxFest

September 14, 2013

I Want You To Use IPv6

Vint Cerf wants YOU to use IPv6

IPv6 Poster

IPv6 infographic

IPv6 Basics

IPv6 Packet Header


from Wikipedia

IPv6 Packet Header

IPv6 Scopes

IPv6 Scopes for Multicast

IPv6 Addresses

A wise co-worker once told me that if you have a handle on IPv6 addresses, you're most of the way to understanding IPv6.

IPv6 Addresses

IPv6 Support

University of Wisconsin at Madison list of operating systems supporting IPv6

IPv6 Performance

IPv6 Performance

Since "prefer v6" goes 6 then 4, there may be a timeout. Web pages may be partly or entirely blank, and IE has it the worst since it doesn't render as pieces arrive.

What's My Address?

hope@calvin$ ifconfig eth0
eth0 Link encap:Ethernet HWaddr 00:14:4F:01:A6:DA 
 inet addr:132.91.236.15 Bcast:132.91.239.255 Mask:255.255.252.0
 inet6 addr: 2001:db8:1000:2001::4/64 Scope:Global
 inet6 addr: fe80::214:4fff:fe01:a6da/64 Scope:Link
 UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
 RX packets:7033253 errors:0 dropped:0 overruns:0 frame:0
 TX packets:3272993 errors:0 dropped:0 overruns:0 carrier:0
 collisions:0 txqueuelen:1000 
 RX bytes:776277559 (740.3 MiB) TX bytes:718801387 (685.5 MiB)

Many Other IPv6 Addresses

IPv6 can have random addresses that change often thanks to RFC 4941 - Privacy Extensions for Stateless Address Autoconfiguration in IPv6 (the update to RFC 3041)

Some Corollaries

IPv4IPv6
ARPNDP
IGMPMLD

Troubleshooting with OSI Layers

IPv4IPv6
1no change
2arpndp is part of ICMPv6
3ping, traceroute,
host
ping6, traceroute6,
host -6
4no change, netstat
7iptablesip6tables

see also University of Wisconsin-Madison's list of network troubleshooting tools

ICMPv6

You can't turn it off!

No Broadcast

IPv6 only uses multicast!

NDP

NDP

ndp and ndisc6

ndp hostname

 -a (all)
 -d (delete)
 -n (numeric, skip name resolution)

hope@calvin$ sudo ndisc6 hobbes6 eth0
Soliciting hobbes6 (2001:db8:1000:2001::5) on eth0...
Target link-layer address: 00:14:4F:01:29:8C
 from 2001:db8:1000:2001::5

ndp alternative

If ndp isn't present, use ip -f inet6 neighbor or ip -6 neighbor show (may use neigh instead)

hope@calvin$ ip -f inet6 neigh | grep -v router
fe80::214:4fff:fe01:298c dev eth0 lladdr 00:14:4f:01:29:8c REACHABLE
2001:db8:1000:2001::5 dev eth0 lladdr 00:14:4f:01:29:8c REACHABLE
hope@calvin$ ip -f inet6 neigh
2001:db8:1000:2001::2 dev eth0 lladdr 00:01:d7:68:fe:c4 router REACHABLE
fe80::214:4fff:fe01:298c dev eth0 lladdr 00:14:4f:01:29:8c REACHABLE
fe80::201:d7ff:fe68:fec4 dev eth0 lladdr 00:01:d7:68:fe:c4 router REACHABLE
2001:db8:1000:2001::1 dev eth0 lladdr 00:01:d7:68:fe:c4 router REACHABLE
2001:db8:1000:2001::3 dev eth0 lladdr 00:01:d7:73:66:c4 router REACHABLE
fe80::201:d7ff:fe73:66c4 dev eth0 lladdr 00:01:d7:73:66:c4 router REACHABLE
2001:db8:1000:2001::5 dev eth0 lladdr 00:14:4f:01:29:8c REACHABLE

prefer v6

When we added IPv6 to calvin's DNS registration, only non-Windows users could ssh in. So I would describe Microsoft's version of "prefer v6" to be "fail badly if IPv6 isn't working" when Linux and Mac failed over to IPv4 silently and quickly under the same circumstances. (My guess is that Mac and Linux listened to the NUD message, and Windows is just a nudnik.)

ping6, traceroute6

I added IPv6 to DNS on calvin. When I went to test it, I discovered I could ping6 and host -6 calvin from hobbes, but not from susie. The first two have static IPv6 addresses and are in the same F5 VLAN, while susie has an automatic IPv6 address (SLAAC) for its VLAN. I also know susie's IPv6 can reach the outside IPv6 world. I suspected the VLAN, so I turned to our Layer 3 tools.

hope@calvin$ host -6 -t AAAA www.mysite.edu calvin6
Using domain server:
Name: calvin6
Address: 2001:db8:1000:2001::4#53
Aliases: 

www.mysite.edu has IPv6 address 2001:db8:1000:2001::11

on calvin6

hope@calvin$ host calvin
calvin.net.mysite.edu has address 132.91.236.15
calvin.net.mysite.edu has IPv6 address 2001:db8:1000:2001::4
hope@calvin$ ifconfig eth0
eth0 Link encap:Ethernet HWaddr 00:14:4F:01:A6:DA
 inet addr:132.91.236.15 Bcast:132.91.239.255 Mask:255.255.252.0
 inet6 addr: 2001:db8:1000:2001::4/64 Scope:Global
 inet6 addr: fe80::214:4fff:fe01:a6da/64 Scope:Link
 UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
 RX packets:53237 errors:0 dropped:0 overruns:0 frame:0
 TX packets:46730 errors:0 dropped:0 overruns:0 carrier:0
 collisions:0 txqueuelen:1000
 RX bytes:4034756 (3.8 MiB) TX bytes:8427890 (8.0 MiB)

on calvin6

hope@calvin$ ping6 -c 4 2001:db8:1000:2001::1
PING 2001:db8:1000:2001::1(2001:db8:1000:2001::1) 56 data bytes
64 bytes from 2001:db8:1000:2001::1: icmp_seq=0 ttl=64 time=0.150 ms
64 bytes from 2001:db8:1000:2001::1: icmp_seq=1 ttl=64 time=0.205 ms
64 bytes from 2001:db8:1000:2001::1: icmp_seq=2 ttl=64 time=0.165 ms
64 bytes from 2001:db8:1000:2001::1: icmp_seq=3 ttl=64 time=0.171 ms

--- 2001:db8:1000:2001::1 ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 2999ms
rtt min/avg/max/mdev = 0.150/0.172/0.205/0.025 ms, pipe 2
hope@calvin$ ping6 -c 4 hobbes
PING hobbes(2001:db8:1000:2001::5) 56 data bytes
64 bytes from 2001:db8:1000:2001::5: icmp_seq=0 ttl=64 time=1.43 ms
64 bytes from 2001:db8:1000:2001::5: icmp_seq=1 ttl=64 time=0.120 ms
64 bytes from 2001:db8:1000:2001::5: icmp_seq=2 ttl=64 time=0.128 ms
64 bytes from 2001:db8:1000:2001::5: icmp_seq=3 ttl=64 time=0.135 ms

--- hobbes ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 3000ms
rtt min/avg/max/mdev = 0.120/0.453/1.432/0.565 ms, pipe 2

hobbes6 to calvin6

hope@hobbes$ ping6 -c 4 calvin
PING calvin(2001:db8:1000:2001::4) 56 data bytes
64 bytes from 2001:db8:1000:2001::4: icmp_seq=0 ttl=64 time=0.125 ms
64 bytes from 2001:db8:1000:2001::4: icmp_seq=1 ttl=64 time=0.134 ms
64 bytes from 2001:db8:1000:2001::4: icmp_seq=2 ttl=64 time=0.104 ms
64 bytes from 2001:db8:1000:2001::4: icmp_seq=3 ttl=64 time=0.114 ms

--- calvin ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 3001ms
rtt min/avg/max/mdev = 0.104/0.119/0.134/0.013 ms, pipe 2

susie to calvin6

hope@susie$ host susie
susie.net.mysite.edu has address 132.90.145.183
hope@susie$ ifconfig eth0
eth0 Link encap:Ethernet HWaddr E4:1F:13:2C:80:18
 inet addr:132.90.145.183 Bcast:132.90.255.255 Mask:255.255.255.0
 inet6 addr: 2001:db8:1000:5001:e61f:13ff:fe2c:8018/64 Scope:Global
 inet6 addr: fe80::e61f:13ff:fe2c:8018/64 Scope:Link
 UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
 RX packets:676169 errors:0 dropped:0 overruns:0 frame:0
 TX packets:84130 errors:0 dropped:0 overruns:0 carrier:0
 collisions:0 txqueuelen:1000
 RX bytes:63870876 (60.9 MiB) TX bytes:10642501 (10.1 MiB)
 Interrupt:169 Memory:92000000-92012800

susie to calvin6

hope@susie$ ping6 -c 4 2001:db8:1000:2001::4
PING 2001:db8:1000:2001::4(2001:db8:1000:2001::4) 56 data bytes 
From 2001:db8:1000:24::1 icmp_seq=0 Destination unreachable: Address unreachable 
From 2001:db8:1000:24::1 icmp_seq=1 Destination unreachable: Address unreachable 
From 2001:db8:1000:24::1 icmp_seq=2 Destination unreachable: Address unreachable 
From 2001:db8:1000:24::1 icmp_seq=3 Destination unreachable: Address unreachable

--- 2001:db8:1000:2001::4 ping statistics ---
4 packets transmitted, 0 received, +4 errors, 100% packet loss, time 3003ms
hope@susie$ ping6 -c 4 calvin
PING calvin(2001:db8:1000:2001::4) 56 data bytes 
From 2001:db8:1000:24::1 icmp_seq=0 Destination unreachable: Address unreachable 
From 2001:db8:1000:24::1 icmp_seq=1 Destination unreachable: Address unreachable 
From 2001:db8:1000:24::1 icmp_seq=2 Destination unreachable: Address unreachable 
From 2001:db8:1000:24::1 icmp_seq=3 Destination unreachable: Address unreachable

--- calvin ping statistics ---
4 packets transmitted, 0 received, +4 errors, 100% packet loss, time 3001ms

susie to calvin6's gateway

hope@susie$ ping6 -c 4 2001:db8:1000:2001::1
PING 2001:db8:1000:2001::1(2001:db8:1000:2001::1) 56 data bytes 
From 2001:db8:1000:24::1 icmp_seq=0 Destination unreachable: Address unreachable 
From 2001:db8:1000:24::1 icmp_seq=1 Destination unreachable: Address unreachable 
From 2001:db8:1000:24::1 icmp_seq=2 Destination unreachable: Address unreachable 
From 2001:db8:1000:24::1 icmp_seq=3 Destination unreachable: Address unreachable

--- 2001:db8:1000:2001::1 ping statistics ---
4 packets transmitted, 0 received, +4 errors, 100% packet loss, time 3001ms

calvin6 to susie

hope@calvin$ ping6 -c 4 2001:db8:1000:5001:e61f:13ff:fe2c:8018
PING
2001:db8:1000:5001:e61f:13ff:fe2c:8018(2001:db8:1000:5001:e61f:13ff:fe2c:8018)
56 data bytes

--- 2001:db8:1000:5001:e61f:13ff:fe2c:8018 ping statistics ---
4 packets transmitted, 0 received, 100% packet loss, time 3000ms

traceroute6 between the two VLANs

hope@susie$ traceroute6 2001:db8:1000:2001::5
traceroute to 2001:db8:1000:2001::5 (2001:db8:1000:2001::5), 30 hops max, 40 byte packets
 1 (2001:db8:1000:5001::1) 0.594 ms 0.616 ms 0.713 ms
 2 (2001:db8:1000:24::1) 0.750 ms 0.837 ms 0.912 ms
 3 (2001:db8:1000:24::1) 0.997 ms !H 1.096 ms !H 1.166 ms !H
hope@susie$
hope@hobbes$ traceroute6 2001:db8:1000:5001:e61f:13ff:fe2c:8018
traceroute to 2001:db8:1000:5001:e61f:13ff:fe2c:8018 (2001:db8:1000:5001:e61f:13ff:fe2c:8018),
 30 hops max, 40 byte packets
 1 (2001:db8:1000:5001:e61f:13ff:fe2c:8018) 0.211 ms 0.202 ms 0.171 ms
 2 (2001:db8:1000:5001:e61f:13ff:fe2c:8018) 0.723 ms 0.621 ms 0.821 ms
 3 * * *
 . . .
30 * * *
hope@hobbes$ 

netstat

hope@calvin$ netstat -an | grep "Proto\|2001"
Proto Recv-Q Send-Q Local Address Foreign Address State 
tcp 0 0 2001:db8:1000:2001::4:2220 2001:db8:1000:5001:221:58302 ESTABLISHED 
tcp 0 0 2001:db8:1000:2001::4:2220 2001:db8:1000:5001:221:58502 ESTABLISHED 
udp 0 0 2001:db8:1000:2001::4:53 :::*  
udp 0 0 2001:db8:1000:2001::4:123 :::*

TCPv6

Most higher-level protocols, like TCP and UDP, are unchanged by IPv6, but some, like FTP and NTP, need specific updates. (ref) The checksum calculation does change for TCP and UDP, though.

ip6tables

hope@calvin$ sudo service ip6tables status
Table: filter
Chain INPUT (policy ACCEPT)
num target prot opt source destination 
1 RH-Firewall-1-INPUT all ::/0 ::/0 

Chain FORWARD (policy ACCEPT)
num target prot opt source destination 
1 RH-Firewall-1-INPUT all ::/0 ::/0 

Chain OUTPUT (policy ACCEPT)
num target prot opt source destination 

Chain RH-Firewall-1-INPUT (2 references)
num target prot opt source destination 
1 ACCEPT all ::/0 ::/0 
2 ACCEPT icmpv6 ::/0 ::/0 
3 ACCEPT esp ::/0 ::/0 
4 ACCEPT ah ::/0 ::/0 

ip6tables

5 ACCEPT udp ::/0 ff02::fb/128 udp dpt:5353 
6 ACCEPT udp ::/0 ::/0 udp dpt:631 
7 ACCEPT tcp ::/0 ::/0 tcp dpt:631 
8 ACCEPT udp ::/0 ::/0 udp dpts:32768:61000 
9 ACCEPT tcp ::/0 ::/0 tcp dpts:32768:61000 flags:!0x16/0x02 
10 ACCEPT tcp ::/0 ::/0 tcp dpt:8080 
11 ACCEPT tcp ::/0 ::/0 tcp dpt:22 
12 REJECT all ::/0 ::/0 reject-with icmp6-adm-prohibited 

or

hope@calvin$ sudo service ip6tables status
Firewall is stopped.

Cookbook

ARIN's wiki page on troubleshooting IPv6 issues

Summary

Conclusion

All of your network troubleshooting instincts are still valid for IPv6, but some of the commands are updated.

Don't fear the

More Information