This chapter describes the how to monitor network performance. The following is a list of the step-by-step instructions in this chapter.How to Check the Response of Hosts on the Network How to Send Packets to Hosts on the Network How to Capture Packets From the Network How to Check the Network Status How to Display NFS Server and Client Statistics Table 30–1 describes the commands that are available for monitoring network performance. Command Description ping Look at the response of hosts on the network. spray Test the reliability of your packet sizes. This command can tell you whether the network is delaying packets or dropping packets. snoop Capture packets from the network and trace the calls from each client to each server. netstat Display network status, including state of the interfaces that are used for TCP/IP traffic, the IP routing table, and the per-protocol statistics for UDP, TCP, ICMP, and IGMP. nfsstat Display a summary of server and client statistics that can be used to identify NFS problems. Check the response of hosts on the network with the ping command.$ ping hostnameIf you suspect a physical problem, you can use ping to find the response time of several hosts on the network. If the response from one host is not what you would expect, you can investigate that host. Physical problems could be caused by the following:Loose cables or connectors Improper grounding No termination Signal reflection For more information about this command, see ping1M.The simplest version of ping sends a single packet to a host on the network. If ping receives the correct response, the command prints the message host is alive. $ ping elvis elvis is aliveWith the s option, ping sends one datagram per second to a host. The command then prints each response and the time that was required for the round trip. An example follows.$ ping -s pluto 64 bytes from pluto (123.456.78.90): icmp_seq=0. time=3.82 ms 64 bytes from pluto (123.456.78.90): icmp_seq=5. time=0.947 ms 64 bytes from pluto (123.456.78.90): icmp_seq=6. time=0.855 ms ^C ----pluto PING Statistics---- 3 packets transmitted, 3 packets received, 0% packet loss round-trip (ms) min/avg/max/sttdev = 0.855/1.87/3.82/1.7 Test the reliability of your packet sizes with the spray command. $ spray [ -c count -d interval -l packet-size] hostnamei countNumber of packets to send. d intervalNumber of microseconds to pause between sending packets. If you do not use a delay, you might deplete the buffers. l packet-sizeIs the packet size. hostnameIs the system to send packets. For more information about this command, see spray1M.The following example sends 100 packets to a host (-c 100), with a packet size of 2048 bytes (-l 2048). The packets are sent with a delay time of 20 microseconds between each burst (-d 20). $ spray -c 100 -d 20 -l 2048 pluto sending 100 packets of length 2048 to pluto ... no packets dropped by pluto 279 packets/sec, 573043 bytes/sec To capture packets from the network and trace the calls from each client to each server, use snoop. This command provides accurate timestamps that enable some network performance problems to be isolated quickly. For more information, see snoop1M.# snoopDropped packets could be caused by insufficient buffer space or an overloaded CPU. To display network status information, such as statistics about the state of network interfaces, routing tables, and various protocols, use the netstat command.$ netstat [-i] [-r] [-s]iDisplays the state of the TCP/IP interfaces rDisplays the IP routing table sDisplays statistics for the UDP, TCP, ICMP, and IGMP protocols For more information, see netstat1M.The following example shows output from the netstat -i command, which displays the state of the interfaces that are used for TCP/IP traffic. $ netstat -i Name Mtu Net/Dest Address Ipkts Ierrs Opkts Oerrs Collis Queue lo0 8232 software localhost 1280 0 1280 0 0 0 eri0 1500 loopback venus 1628480 0 347070 16 39354 0This display shows the number of packets that a machine has transmitted and has received on each interface. A machine with active network traffic should show both Ipkts and Opkts continually increasing. Calculate the network collisions rate by dividing the number of collision counts (Collis) by the number of out packets (Opkts). In the previous example, the collision rate is 11 percent. A network-wide collision rate that is greater than 5 to 10 percent can indicate a problem. Calculate the error rate for the input packets by dividing the number of input errors by the total number of input packets (Ierrs/Ipkts). The error rate for the output packets is the number of output errors divided by the total number of output packets (Oerrs/Opkts). If the input error rate is high, at over 0.25 percent, the host might be dropping packets. The following example shows output from the netstat -s command, which displays the per-protocol statistics for the UDP, TCP, ICMP, and IGMP protocols.UDP udpInDatagrams =196543 udpInErrors = 0 udpOutDatagrams =187820 TCP tcpRtoAlgorithm = 4 tcpRtoMin = 200 tcpRtoMax = 60000 tcpMaxConn = -1 tcpActiveOpens = 26952 tcpPassiveOpens = 420 tcpAttemptFails = 1133 tcpEstabResets = 9 tcpCurrEstab = 31 tcpOutSegs =3957636 tcpOutDataSegs =2731494 tcpOutDataBytes =1865269594 tcpRetransSegs = 36186 tcpRetransBytes =3762520 tcpOutAck =1225849 tcpOutAckDelayed =165044 tcpOutUrg = 7 tcpOutWinUpdate = 315 tcpOutWinProbe = 0 tcpOutControl = 56588 tcpOutRsts = 803 tcpOutFastRetrans = 741 tcpInSegs =4587678 tcpInAckSegs =2087448 tcpInAckBytes =1865292802 tcpInDupAck =109461 tcpInAckUnsent = 0 tcpInInorderSegs =3877639 tcpInInorderBytes =-598404107 tcpInUnorderSegs = 14756 tcpInUnorderBytes =17985602 tcpInDupSegs = 34 tcpInDupBytes = 32759 tcpInPartDupSegs = 212 tcpInPartDupBytes =134800 tcpInPastWinSegs = 0 tcpInPastWinBytes = 0 tcpInWinProbe = 456 tcpInWinUpdate = 0 tcpInClosed = 99 tcpRttNoUpdate = 6862 tcpRttUpdate =435097 tcpTimRetrans = 15065 tcpTimRetransDrop = 67 tcpTimKeepalive = 763 tcpTimKeepaliveProbe= 1 tcpTimKeepaliveDrop = 0 IP ipForwarding = 2 ipDefaultTTL = 255 ipInReceives =11757234 ipInHdrErrors = 0 ipInAddrErrors = 0 ipInCksumErrs = 0 ipForwDatagrams = 0 ipForwProhibits = 0 ipInUnknownProtos = 0 ipInDiscards = 0 ipInDelivers =4784901 ipOutRequests =4195180 ipOutDiscards = 0 ipOutNoRoutes = 0 ipReasmTimeout = 60 ipReasmReqds = 8723 ipReasmOKs = 7565 ipReasmFails = 1158 ipReasmDuplicates = 7 ipReasmPartDups = 0 ipFragOKs = 19938 ipFragFails = 0 ipFragCreates =116953 ipRoutingDiscards = 0 tcpInErrs = 0 udpNoPorts =6426577 udpInCksumErrs = 0 udpInOverflows = 473 rawipInOverflows = 0 ICMP icmpInMsgs =490338 icmpInErrors = 0 icmpInCksumErrs = 0 icmpInUnknowns = 0 icmpInDestUnreachs = 618 icmpInTimeExcds = 314 icmpInParmProbs = 0 icmpInSrcQuenchs = 0 icmpInRedirects = 313 icmpInBadRedirects = 5 icmpInEchos = 477 icmpInEchoReps = 20 icmpInTimestamps = 0 icmpInTimestampReps = 0 icmpInAddrMasks = 0 icmpInAddrMaskReps = 0 icmpInFragNeeded = 0 icmpOutMsgs = 827 icmpOutDrops = 103 icmpOutErrors = 0 icmpOutDestUnreachs = 94 icmpOutTimeExcds = 256 icmpOutParmProbs = 0 icmpOutSrcQuenchs = 0 icmpOutRedirects = 0 icmpOutEchos = 0 icmpOutEchoReps = 477 icmpOutTimestamps = 0 icmpOutTimestampReps= 0 icmpOutAddrMasks = 0 icmpOutAddrMaskReps = 0 icmpOutFragNeeded = 0 icmpInOverflows = 0 IGMP: 0 messages received 0 messages received with too few bytes 0 messages received with bad checksum 0 membership queries received 0 membership queries received with invalid field(s) 0 membership reports received 0 membership reports received with invalid field(s) 0 membership reports received for groups to which we belong 0 membership reports sentThe following example shows output from the netstat r command, which displays the IP routing table. Routing Table: Destination Gateway Flags Ref Use Interface ------------------ -------------------- ----- ----- ------ --------- localhost localhost UH 0 2817 lo0 earth-bb pluto U 3 14293 eri0 224.0.0.0 pluto U 3 0 eri0 default mars-gate UG 0 14142 The fields in the netstat r report are described in Table 30–2. Field Name Description Flags UGHD The route is up.The route is through a gateway.The route is to a host.The route was dynamically created by using a redirect. Ref Shows the current number of routes that share the same link layer. Use Indicates the number of packets that were sent out. Interface Lists the network interface that is used for the route. The NFS distributed file service uses a remote procedure call (RPC) facility that translates local commands into requests for the remote host. The remote procedure calls are synchronous. The client application is blocked or suspended until the server has completed the call and has returned the results. One of the major factors that affects NFS performance is the retransmission rate. If the file server cannot respond to a client's request, the client retransmits the request a specified number of times before the client quits. Each retransmission imposes system overhead and increases network traffic. Excessive retransmissions can cause network performance problems. If the retransmission rate is high, you could look for the following:Overloaded servers that complete requests too slowly An Ethernet interface that is dropping packets Network congestion, which slows the packet transmission Table 30–3 describes the nfsstat options to display client and server statistics.Command Display nfsstat -c Client statistics nfsstat -s Server statistics netstat -m Network statistics for each file system Use nfsstat -c to show client statistics, and nfsstat -s to show server statistics. Use netstat -m to display network statistics for each file system. For more information, see nfsstat1M. The following example displays RPC and NFS data for the client pluto. $ nfsstat -c Client rpc: Connection oriented: calls badcalls badxids timeouts newcreds badverfs timers 1595799 1511 59 297 0 0 0 cantconn nomem interrupts 1198 0 7 Connectionless: calls badcalls retrans badxids timeouts newcreds badverfs 80785 3135 25029 193 9543 0 0 timers nomem cantsend 17399 0 0 Client nfs: calls badcalls clgets cltoomany 1640097 3112 1640097 0 Version 2: (46366 calls) null getattr setattr root lookup readlink read 0 0% 6589 14% 2202 4% 0 0% 11506 24% 0 0% 7654 16% wrcache write create remove rename link symlink 0 0% 13297 28% 1081 2% 0 0% 0 0% 0 0% 0 0% mkdir rmdir readdir statfs 24 0% 0 0% 906 1% 3107 6% Version 3: (1585571 calls) null getattr setattr lookup access readlink read 0 0% 508406 32% 10209 0% 263441 16% 400845 25% 3065 0% 117959 7% write create mkdir symlink mknod remove rmdir 69201 4% 7615 0% 42 0% 16 0% 0 0% 7875 0% 51 0% rename link readdir readdir+ fsstat fsinfo pathconf 929 0% 597 0% 3986 0% 185145 11% 942 0% 300 0% 583 0% commit 4364 0% Client nfs_acl: Version 2: (3105 calls) null getacl setacl getattr access 0 0% 0 0% 0 0% 3105 100% 0 0% Version 3: (5055 calls) null getacl setacl 0 0% 5055 100% 0 0% The output of the nfsstat -c command is described in Table 30–4.Field Description calls The total number of calls that were sent. badcalls The total number of calls that were rejected by RPC. retrans The total number of retransmissions. For this client, the number of retransmissions is less than 1 percent, or approximately 10 timeouts out of 6888 calls. These retransmissions might be caused by temporary failures. Higher rates might indicate a problem. badxid The number of times that a duplicate acknowledgment was received for a single NFS request. timeout The number of calls that timed out. wait The number of times a call had to wait because no client handle was available. newcred The number of times the authentication information had to be refreshed. timers The number of times the time-out value was greater than or equal to the specified time-out value for a call. readlink The number of times a read was made to a symbolic link. If this number is high, at over 10 percent, then there could be too many symbolic links. The following example shows output from the nfsstat -m command.pluto$ nfsstat -m /usr/man from pluto:/export/svr4/man Flags: vers=2,proto=udp,auth=unix,hard,intr,dynamic, rsize=8192, wsize=8192,retrans=5 Lookups: srtt=13 (32ms), dev=10 (50ms), cur=6 (120ms) All: srtt=13 (32ms), dev=10 (50ms), cur=6 (120ms)This output of the nfsstat -m command, which is displayed in milliseconds, is described in Table 30–5.Field Description srtt The smoothed average of the round-trip times dev The average deviations cur The current “expected” response time If you suspect that the hardware components of your network are creating problems, you need to look closely at the cabling and connectors.