Too much TCP retransmitted and TCP duplicate on server Oracle Solaris 10

I have problem with oracle solaris 10 running on oracle sparc T4-2 server.
Os information: 5.10 Generic_150400-03 sun4v sparc sun4v
Output from tcpstat.d script

  TCP bytes:     out   outRetrans           in        inDup    inUnorder
             6833763         7300        98884            0            0
             1974525          864        63993            0            0
             1975044            0        44712            0            0
             2601126          120        50980            0            0
             4812048           39        95983            0            0
             4150302          547        74640           14            0
              916926            0        46583           32            0
             3254534            0       344162            0            0
             7634514          175        92269            0            0
             1470737            0        56622            0            0
             3302183          280        91304            0            0
             2285685          144        65818            0            0
             4516005           63       104704            0            0
             5492292            0       104575            0            0
             2940265          864       152483            0            0
             5144902            0       124908            0            0
             2496748            0        88227            0            0
             1713836            0       116517           65            0
             2692040          136        95233            0            0
             3256049          136       118073            0            0

Snoop information on server: Please see attach picture.
netstat -sP tcp information

TCP     tcpRtoAlgorithm     =     4     tcpRtoMin           =   400
        tcpRtoMax           = 60000     tcpMaxConn          =    -1
        tcpActiveOpens      =183824     tcpPassiveOpens     =184227
        tcpAttemptFails     =  2110     tcpEstabResets      = 73884
        tcpCurrEstab        =   325     tcpOutSegs          =6413525798
        tcpOutDataSegs      =210063122  tcpOutDataBytes     =3129665400
        tcpRetransSegs      =7724022    tcpRetransBytes     =611738266
        tcpOutAck           =1685731391 tcpOutAckDelayed    =263533955
        tcpOutUrg           = 12625     tcpOutWinUpdate     =  6951
        tcpOutWinProbe      =  5267     tcpOutControl       =754008
        tcpOutRsts          = 24425     tcpOutFastRetrans   =  5287
        tcpInSegs           =4836233979
        tcpInAckSegs        =296120770  tcpInAckBytes       =1389371405
        tcpInDupAck         =15641972   tcpInAckUnsent      =   474
        tcpInInorderSegs    =2044378119 tcpInInorderBytes   =2246362655
        tcpInUnorderSegs    =146886     tcpInUnorderBytes   =66009739
        tcpInDupSegs        =1384221    tcpInDupBytes       =128124335
        tcpInPartDupSegs    =   708     tcpInPartDupBytes   =136252
        tcpInPastWinSegs    =  5873     tcpInPastWinBytes   =3770680883
        tcpInWinProbe       =   170     tcpInWinUpdate      =  4739
        tcpInClosed         =  8220     tcpRttNoUpdate      =3314699518
        tcpRttUpdate        =1285937982 tcpTimRetrans       =5164257
        tcpTimRetransDrop   =  9274     tcpTimKeepalive     =147590
        tcpTimKeepaliveProbe= 24761     tcpTimKeepaliveDrop =  2037
        tcpListenDrop       =  2209     tcpListenDropQ0     =     0
        tcpHalfOpenDrop     =     0     tcpOutSackRetrans   =2574473

When i ping test from server to another server, sometimes server hold icmp packet 10 -> 30s same times with too much tcp retransmitt and tcp duplicate on server.

Fri Apr  4 14:49:47 ICT 2014: 64 bytes from 172.16.11.13: icmp_seq=853. time=0.547 ms
Fri Apr  4 14:49:48 ICT 2014: 64 bytes from 172.16.11.13: icmp_seq=854. time=5.22 ms
Fri Apr  4 14:49:49 ICT 2014: 64 bytes from 172.16.11.13: icmp_seq=855. time=0.513 ms
Fri Apr  4 14:49:50 ICT 2014: 64 bytes from 172.16.11.13: icmp_seq=856. time=0.618 ms
Fri Apr  4 14:49:59 ICT 2014: 64 bytes from 172.16.11.13: icmp_seq=857. time=8.12e+03 ms
Fri Apr  4 14:49:59 ICT 2014: 64 bytes from 172.16.11.13: icmp_seq=858. time=7.12e+03 ms
Fri Apr  4 14:49:59 ICT 2014: 64 bytes from 172.16.11.13: icmp_seq=859. time=6.13e+03 ms
Fri Apr  4 14:49:59 ICT 2014: 64 bytes from 172.16.11.13: icmp_seq=860. time=5.13e+03 ms
Fri Apr  4 14:49:59 ICT 2014: 64 bytes from 172.16.11.13: icmp_seq=861. time=4.13e+03 ms
Fri Apr  4 14:49:59 ICT 2014: 64 bytes from 172.16.11.13: icmp_seq=862. time=3.13e+03 ms
Fri Apr  4 14:49:59 ICT 2014: 64 bytes from 172.16.11.13: icmp_seq=863. time=2.13e+03 ms
Fri Apr  4 14:49:59 ICT 2014: 64 bytes from 172.16.11.13: icmp_seq=864. time=1.13e+03 ms
Fri Apr  4 14:49:59 ICT 2014: 64 bytes from 172.16.11.13: icmp_seq=865. time=130. ms
Fri Apr  4 14:50:00 ICT 2014: 64 bytes from 172.16.11.13: icmp_seq=866. time=0.606 ms
Fri Apr  4 14:50:01 ICT 2014: 64 bytes from 172.16.11.13: icmp_seq=867. time=10.3 ms

What's the network between the two?

1 Like

Server connect to coreswitch, all data from server go through checkpoint firewall. When i capture packet from server, example using ping command: icmp request and reply done well but ping command delay 10->30s to show result