Home
Personal
Unix
Programming
Networking
Cisco
 Songbook
 Programming
 Tools
 Basic Cisco Router Security
 Getting contact info on the Internet
 Why Mail fails
 Basic Network Troubleshooting
 Network and System Monitoring Primers
 Documents
Reporting
Weblog
CityRail
BOM pictures
Other projects
Contact me
               
   

Basic network troubleshooting

This document describes how to do basic network troubleshooting. It shows the tools and the (un)common output with them.

Jargon

  • remote host: the machine you're trying to find information about.
  • alias: a different name for a machine.
  • ttl: Time To Live value
  • hop: A router in the path of an IP packet from your machine to a remote host.

Ping

Ping is a basic tool to see if there is IP connectitivy towards the remote host. The basic syntax is ping <hostname>, for example ping www.mavetju.org. It will print a line for every answer it received. If you ping a remote host and you receive an answer from it then the machine is reachable on IP level.

Reachable hosts

If everything is okay it will display: 

 1. [~] edwin@k7>ping www.mavetju.org
 2. PING topaz.mdcc.cx (212.204.230.141): 56 data bytes
 3. 64 bytes from 212.204.230.141: icmp_seq=0 ttl=237 time=681.722 ms
 4. 64 bytes from 212.204.230.141: icmp_seq=1 ttl=237 time=550.007 ms
 5. 64 bytes from 212.204.230.141: icmp_seq=2 ttl=237 time=790.802 ms
 6. 64 bytes from 212.204.230.141: icmp_seq=3 ttl=237 time=699.964 ms
 7. ^C
 8. --- topaz.mdcc.cx ping statistics ---
 9. 5 packets transmitted, 4 packets received, 20% packet loss
10. round-trip min/avg/max/stddev = 540.111/652.877/790.802/98.965 ms

The following information can be extracted from the output.

  • Line 2:
    • Although you try to ping www.mavetju.org, it actually tries to ping topaz.mdcc.cx. That's because www.mavetju.org is hosted on that machine.
  • Lines 3 to 6:
    • It says it received answer from 212.204.230.141. Sometimes it receives answers from other machines. See later examples about these situations.
    • The TTL of the received packet is 237. The TTL is an 8 bit digit, which means it can be from 0 to 255. At start it's set tp 255 and every hop on the way towards your machine decreases it by one. So this it took 255 - 237 = 18 hops to get from topaz.mdcc.cx towards my machine.
    • The time it took to receive an answer since the original request. This is an indication for how reachable the remote host is. The reason for these huge numbers for me is because I'm going from Australia towards the Netherlands.
  • Line 9 and 10:
    • At the end it shows how many packets were send and how many were received. This is an indication for the reliability of the line.
    • and it shows the fastests, avaraged, slowests and deviation of the times.

Blocked hosts

Sometimes people don't want their hosts to be pinged and have configured their routers to block ping-packets. You might see such an output then: 

 1. [~] edwin@k7>ping www.mavetju.org
 2. PING topaz.mdcc.cx (212.204.230.141): 56 data bytes
 3. 36 bytes from gateway.widexs.net (212.204.214.161): Communication prohibited by filter
 4. Vr HL TOS  Len   ID Flg  off TTL Pro  cks      Src      Dst
 5.  4  5  00 5400 d61f   0 0000  f1  01 ba99 203.173.128.219  212.204.230.141

The following information can be extracted from the output:

  • Line 3:
    • The router filtering the traffic is gateway.widexs.net
    • The reason why it replied was Communication prohibited by filter. That means that the router was configured to block ping-packets towards that machine.
  • Lines 4 and 5:
    • This is a dump of the IP header and gives some information regarding the TTL and the source and destination IP addresses.

Unreachable hosts and networks

Sometimes when an ISP has problems with its connectivity towards the internet you see messages regarding Destination Host Unreachable or Destination Network Unreachable. This means that the routers on the internet don't know where to find that IP address. 

 1. [~] edwin@k7>ping www.mavetju.org
 2. PING topaz.mdcc.cx (212.204.230.141): 56 data bytes
 3. 36 bytes from 107.ATM2-0-0.GW2.SYD2.ALTER.NET (203.166.91.53): Destination Host Unreachable
 4. Vr HL TOS  Len   ID Flg  off TTL Pro  cks      Src      Dst
 5.  4  5  00 5400 dd1d   0 0000  fc  01 90fc 203.173.128.219  212.204.230.141

107.ATM2-0-0.GW2.SYD2.ALTER.NET was the machine which didn't know where to find 212.204.230.141 anymore.

Unreachable remote hosts

If a remote host is unreachable, ping will not display any output. 

 1. [~] edwin@k7>ping www.mavetju.org
 2. PING topaz.mdcc.cx (212.204.230.141): 56 data bytes
 3. ^C
 4. --- topaz.mdcc.cx ping statistics ---
 5. 56 packets transmitted, 0 packets received, 100% packet loss

It didn't print anything and at the end it said: 100% packet loss. That means that the machine is unreachable. It didn't send any usefull to debug. See later example with traceroute how to investigate further.

Traceroute

Traceroute is used to find out the route IP packets use to come to a remote host. The basic syntax is traceroute <host>, for example traceroute www.mavetju.org. It will print one line per hop.

Reachable hosts

If everything is okay it will display: 

 1. [~] edwin@k7>traceroute www.mavetju.org
 2. traceroute to topaz.mdcc.cx (212.204.230.141), 30 hops max, 40 byte packets
 3.  1  tnt1.syd.ihug.com.au (203.56.8.99)  129.419 ms  147.075 ms  149.916 ms
 4.  2  feth5-0-0-tig-aus-syd-1.ihug.net (203.56.8.254)  140.499 ms  135.116 ms  139.620 ms
 5.  3  107.ATM2-0-0.GW2.SYD2.ALTER.NET (203.166.91.53)  299.262 ms  287.410 ms  249.638 ms
 6.  4  322.at-3-0-0.XR2.SYD2.Alter.Net (210.80.3.113)  229.544 ms  257.976 ms  239.899 ms
 7.  5  so-6-0-0.TR1.SYD2.ALTER.NET (210.80.51.249)  269.725 ms  208.059 ms  189.571 ms
 8.  6  296.ATM5-0.IR1.SAC2.ALTER.NET (210.80.51.181)  340.459 ms  366.554 ms  368.856 ms
 9.  7  POS3-0.IR1.SAC1.ALTER.NET (137.39.31.194)  451.491 ms  525.386 ms  479.895 ms
10.  8  120.at-5-1-0.TR1.SAC1.ALTER.NET (152.63.11.62)  469.945 ms  515.503 ms  479.948 ms
11.  9  0.so-7-0-0.XR1.SAC1.ALTER.NET (146.188.144.145)  530.383 ms  537.136 ms  539.722 ms
12. 10  185.ATM7-0.BR1.SAC1.ALTER.NET (152.63.51.61)  559.403 ms  507.996 ms  459.614 ms
13. 11  204.255.168.86 (204.255.168.86)  499.790 ms  488.737 ms
14. 12  acr2-loopback.NewYorknyr.cw.net (206.24.194.62)  579.550 ms  588.881 ms  590.048 ms
15. 13  bcr2-so-6-0-0.Amsterdam.cw.net (206.24.193.226)  619.319 ms  658.429 ms  619.892 ms
16. 14  zcr2-so-1-0-0.Amsterdamamt.cw.net (208.173.209.198)  698.926 ms  579.023 ms  598.942 ms
17. 15  zar1-ge-1-3-0.Amsterdamamt.cw.net (195.10.7.59)  730.086 ms  649.050 ms  609.453 ms
18. 16  cable-and-wireless-internal-isp.Amsterdamamt.cw.net (195.10.34.34)  618.933 ms  558.542 ms  660.020 ms
19. 17  so4-0.hfd1.widexs.net (212.204.214.161)  719.343 ms  648.132 ms  638.972 ms
20. 18  topaz.mdcc.cx (212.204.230.141)  557.982 ms  608.827 ms  599.026 ms

The following information can be extracted from the output.

  • Line 2:
    • Although www.mavetju.org is specified, it actually tries to reach topaz.mdcc.cx. That's because www.mavetju.org is hosted on that machine.
  • Line 3:
    • That's the first hop towards the remote host, it's also the default gateway of your machine.
  • Line 11:
    • This IP address didn't have a reverse lookup.
  • Line 20:
    • The remote host itself.

Missing parts

Sometimes a part of the network doesn't give information. Then would be the result then: 

 1. 11  204.255.168.86 (204.255.168.86)  499.790 ms  488.737 ms
 2. 12  * * *
 3. 13  * * *
 4. 14  zcr2-so-1-0-0.Amsterdamamt.cw.net (208.173.209.198)  698.926 ms  579.023 ms  598.942 ms

Here hop 12 and 13 didn't return any information. This can be because the ISP has disabled sending of this kind of information on outside his own network.

Host does not exist on the LAN

If a host does not exist on the LAN, you will get *'s after the last interface up to the end: 

 1. 17  so4-0.hfd1.widexs.net (212.204.214.161)  568.985 ms  548.131 ms  549.940 ms
 2. 18  * * *
 3. 19  * * *
 4. 20  * * *
    [...]
 5. 31  * * *
 6. 32  * * *

Here the router at widexs.net does still exist, but the remote host on the LAN doesn't give an answer. It's probably turned off.

Access denied by a filter

Sometimes people don't want their hosts to be traced and have configured their routers to block traceroute-packets. You might see such an output then: 

 1. 17  so4-0.hfd1.widexs.net (212.204.214.161)  719.343 ms  648.132 ms  638.972 ms
 2. 18  so4-0.hfd1.widexs.net (212.204.214.161)  719.343 ms !X  648.132 ms !X  638.972 ms !X

The !X means that the router doesn't allow traceroute towards that remote host.

Network is unknown

If the network of the remote host isn't know on a router, you will see a !N. 

 1. [~] edwin@k7>traceroute www.mavetju.org
 2. traceroute to topaz.mdcc.cx (212.204.230.141), 30 hops max, 40 byte packets
 3.  1  tnt1.syd.ihug.com.au (203.56.8.99)  129.419 ms  147.075 ms  149.916 ms
 4.  2  feth5-0-0-tig-aus-syd-1.ihug.net (203.56.8.254)  140.499 ms  135.116 ms  139.620 ms
 5.  3  feth5-0-0-tig-aus-syd-1.ihug.net (203.56.8.254)  140.499 ms !N  135.116 ms !N  139.620 ms !N

At this output the last host didn't know where to to forward the packets for the remote host and returned a "Network Unreachable" message.

Host is unknown

If the host can't be found by its router (after it has done its ARP requests), it might send a message back and you will see a !H. 

 1. [~] edwin@k7>traceroute www.mavetju.org
 2. traceroute to topaz.mdcc.cx (212.204.230.141), 30 hops max, 40 byte packets
 3.  1  tnt1.syd.ihug.com.au (203.56.8.99)  129.419 ms  147.075 ms  149.916 ms
 4.  2  feth5-0-0-tig-aus-syd-1.ihug.net (203.56.8.254)  140.499 ms  135.116 ms  139.620 ms
 5.  3  107.ATM2-0-0.GW2.SYD2.ALTER.NET (203.166.91.53)  299.262 ms  287.410 ms  249.638 ms
 6.  [...]
 7. 17  so4-0.hfd1.widexs.net (212.204.214.161)  719.343 ms  648.132 ms  638.972 ms
 9. 18  so4-0.hfd1.widexs.net (212.204.214.161)  719.343 ms !H  648.132 ms !H  638.972 ms !H

At this output the last router couldn't deliver the IP packet to the host (because it didn't get an answer on the ARP request) and returned a "Host Unreachable" message.

Telnet

When the network has confirmed to be allright, it's time to check the remote host itself. For example if the remote host is running a webserver is should allow a TCP session to port 80.

Reachable services

When a remote host is running a webserver it should allow a TCP session to port 80. 

 1. [~] edwin@k7>telnet www.mavetju.org 80
 2. Trying 212.204.230.141...
 3. Connected to topaz.mdcc.cx.
 4. Escape character is '^]'.

This remote host has a service running on port 80.

Unreachable services

When a remote host isn't running a service, it will refuse the TCP connection. 

 1. [~] edwin@k7>telnet www.mavetju.org 80
 2. Trying 212.204.230.141...
 3. telnet: connect to address 212.204.230.141: Connection refused
 4. telnet: Unable to connect to remote host

Broken services

Sometimes it happens that a service is running but that it doesn't work (for example the webserver is running but it is broken). The setup of the TCP session will timeout then. 

 1. [~] edwin@k7>telnet www.mavetju.org 80
 2. Trying 212.204.230.141...
 3. telnet: connect to address 212.204.230.141: Operation timed out
 4. telnet: Unable to connect to remote host
       
               
               
$Id: basicnetworktroubleshooting.php,v 1.7 2002/10/25 09:01:45 mavetju Exp $