Application showing "Connection Time Out" very often

UNIX for Advanced & Expert Users
1

Hi,

We have a web serveer running on Apache-Tomcat-6.0.32. For 2 months continuously, we noticed that the app on this server setup has been frequently sending out intermittent "Connection Time Out" issues.
On the OS side, I investigated a lot of things,

1) tcpdump : did not find anything suspicious (no large flooding of suspicious IP, etc)
2) In the apache access_log nothing suspicious either.
3) netstat -an, tulpn : No timeout/dead connection
4) telnet to port 80 is OK

Thus, my teammate concluded that it could be a bug in the tomcat version 6.0.32 itself. Is this possible/true? How can we confirm this?
We use "Uptime" as a monitoring tool, and it sends updates every 3 mins. Whereas other monitoring tools send alerts every 5 mins or more, and do not show the "Connection Time Out" problem.
Is "Uptime" an unstable monitoring tool?

2

What about on the network itself?
We dont know what your Tomcat is doing either... is the server just the application server, the data is elsewhere? ( could explain the message then...) etc...
We need to know a bit more in order to help you, like knowing what architecture, OS, RAM, net connections etc...

1 Like
3

Thank you for the response.
What do I check on the network itself?
Yes, the problematic server just hosts the application. The data (I assume the this must be the database) runs on a different server.

The network connections to the server (I used netstat) :

[root@app6 aigini]# netstat -tulpn
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address               Foreign Address             State       PID/Program name
tcp        0      0 192.168.140.228:10050       0.0.0.0:*                   LISTEN      30651/zabbix_agentd
tcp        0      0 0.0.0.0:111                 0.0.0.0:*                   LISTEN      15570/rpcbind
tcp        0      0 192.168.140.228:80          0.0.0.0:*                   LISTEN      5553/httpd
tcp        0      0 0.0.0.0:22                  0.0.0.0:*                   LISTEN      5014/sshd
tcp        0      0 192.168.140.228:443         0.0.0.0:*                   LISTEN      5553/httpd
tcp        0      0 127.0.0.1:2812              0.0.0.0:*                   LISTEN      16476/monit
tcp        0      0 0.0.0.0:35772               0.0.0.0:*                   LISTEN      27416/rpc.statd
tcp        0      0 ::ffff:127.0.0.1:8005       :::*                        LISTEN      16690/java
tcp        0      0 :::8009                     :::*                        LISTEN      16690/java
tcp        0      0 ::ffff:127.0.0.1:8205       :::*                        LISTEN      17058/java
tcp        0      0 :::111                      :::*                        LISTEN      15570/rpcbind
tcp        0      0 ::ffff:127.0.0.1:8305       :::*                        LISTEN      22557/java
tcp        0      0 :::8209                     :::*                        LISTEN      22557/java
tcp        0      0 ::ffff:127.0.0.1:8405       :::*                        LISTEN      16176/java
tcp        0      0 :::8309                     :::*                        LISTEN      17058/java
tcp        0      0 :::22                       :::*                        LISTEN      5014/sshd
tcp        0      0 :::8409                     :::*                        LISTEN      16176/java
tcp        0      0 ::ffff:127.0.0.1:8505       :::*                        LISTEN      17520/java
tcp        0      0 :::8509                     :::*                        LISTEN      17520/java
tcp        0      0 :::56510                    :::*                        LISTEN      27416/rpc.statd
udp        0      0 0.0.0.0:51285               0.0.0.0:*                               27416/rpc.statd
udp        0      0 0.0.0.0:111                 0.0.0.0:*                               15570/rpcbind
udp        0      0 127.0.0.1:880               0.0.0.0:*                               27416/rpc.statd
udp        0      0 0.0.0.0:631                 0.0.0.0:*                               15639/portreserve
udp        0      0 192.168.140.228:123         0.0.0.0:*                               1485/ntpd
udp        0      0 127.0.0.1:123               0.0.0.0:*                               1485/ntpd
udp        0      0 0.0.0.0:123                 0.0.0.0:*                               1485/ntpd
udp        0      0 0.0.0.0:905                 0.0.0.0:*                               15570/rpcbind
udp        0      0 :::50649                    :::*                                    27416/rpc.statd
udp        0      0 :::33384                    :::*                                    22557/java
udp        0      0 :::111                      :::*                                    15570/rpcbind
udp        0      0 fe80::7a2b:cbff:fe78:e99:123 :::*                                    1485/ntpd
udp        0      0 ::1:123                     :::*                                    1485/ntpd
udp        0      0 :::123                      :::*                                    1485/ntpd
udp        0      0 :::905                      :::*                                    15570/rpcbind
udp        0      0 :::53524                    :::*                                    22557/java
[root@app6 aigini]#

Architecture :

[root@app6 aigini]# uname -a
Linux app6.int.myeg.com.my 2.6.32-279.el6.i686 #1 SMP Fri Jun 22 10:59:55 UTC 2012 i686 i686 i386 GNU/Linux
[root@app6 aigini]#

OS :

[root@app6 aigini]# cat /etc/redhat-release
CentOS release 6.9 (Final)
[root@app6 aigini]#

Memory :

[root@app6 aigini]# free -m
             total       used       free     shared    buffers     cached
Mem:          7978       7701        277          3        481       1836
-/+ buffers/cache:       5382       2596
Swap:         8111       1286       6825
[root@app6 aigini]#
4

check your host. As mentioned, most of the time the issue is with the TCP host.