Port hang issue in AIX Version 5.3

On AIX platform we are having Weblogic 8.1 as the middleware for an application.

The application is deployed in Weblogic on AIX platform and the server is having 2 dedicated listening ports for any incoming requests.

Output for netstat -an command for the port 30001 is as follows:-

tcp4 0 0 X.X.X.X.30001 Z.Z.Z.Z.20501 ESTABLISHED
tcp4 0 0 X.X.X.X.30001 Z.Z.Z.Z.47428 ESTABLISHED
tcp4 0 0 X.X.X.X.30001 Z.Z.Z.Z.48077 ESTABLISHED
tcp 0 0 .30001 *. LISTEN
tcp4 0 0 X.X.X.X.30001 Z.Z.Z.Z.52664 ESTABLISHED
tcp4 0 0 X.X.X.X.30001 Z.Z.Z.Z.53170 ESTABLISHED
tcp4 0 0 X.X.X.X.30001 Y.Y.Y.Y.55643 ESTABLISHED
tcp4 0 0 X.X.X.X.30001 Y.Y.Y.Y.55667 ESTABLISHED
tcp4 0 0 X.X.X.X.30001 Y.Y.Y.Y.55668 ESTABLISHED
tcp4 0 0 X.X.X.X.30001 Y.Y.Y.Y.55669 ESTABLISHED
tcp4 0 0 X.X.X.X.30001 Y.Y.Y.Y.55680 ESTABLISHED
tcp4 0 0 X.X.X.X.30001 Y.Y.Y.Y.55691 ESTABLISHED
tcp4 0 0 X.X.X.X.30001 Y.Y.Y.Y.55694 ESTABLISHED
tcp4 0 0 X.X.X.X.30001 Y.Y.Y.Y.55703 ESTABLISHED
tcp4 0 0 X.X.X.X.30001 Y.Y.Y.Y.55719 ESTABLISHED
tcp4 0 0 X.X.X.X.30001 Y.Y.Y.Y.55753 ESTABLISHED
tcp4 0 0 X.X.X.X.30001 Y.Y.Y.Y.55755 ESTABLISHED
tcp4 0 0 X.X.X.X.30001 Y.Y.Y.Y.55758 ESTABLISHED
tcp4 0 0 X.X.X.X.30001 Y.Y.Y.Y.55761 ESTABLISHED
tcp4 0 0 X.X.X.X.30001 Y.Y.Y.Y.55762 ESTABLISHED
tcp4 0 0 X.X.X.X.30001 Y.Y.Y.Y.55781 ESTABLISHED
tcp4 0 0 X.X.X.X.30001 Z.Z.Z.Z.54436 ESTABLISHED
tcp4 0 0 X.X.X.X.30001 Z.Z.Z.Z.54891 ESTABLISHED
tcp4 0 0 X.X.X.X.30001 Z.Z.Z.Z.55333 ESTABLISHED
tcp4 0 0 X.X.X.X.30001 Z.Z.Z.Z.31796 ESTABLISHED
tcp4 0 0 X.X.X.X.30001 Z.Z.Z.Z.31979 ESTABLISHED
tcp4 0 0 X.X.X.X.30001 Z.Z.Z.Z.32428 ESTABLISHED
tcp4 0 0 X.X.X.X.30001 Z.Z.Z.Z.59195 ESTABLISHED
tcp4 0 0 X.X.X.X.30001 Z.Z.Z.Z.59314 ESTABLISHED
tcp4 0 0 X.X.X.X.30001 Z.Z.Z.Z.10405 ESTABLISHED
tcp4 0 0 X.X.X.X.30001 Z.Z.Z.Z.59457 ESTABLISHED
tcp4 0 0 X.X.X.X.30001 Z.Z.Z.Z.12872 ESTABLISHED
tcp4 0 0 X.X.X.X.30001 Z.Z.Z.Z.12943 ESTABLISHED
tcp4 0 0 X.X.X.X.30001 Z.Z.Z.Z.39168 ESTABLISHED
tcp4 0 0 X.X.X.X.30001 Z.Z.Z.Z.42803 ESTABLISHED

With time the no. of established connection get increased and as soon as the count reaches more than 300 the port gets hanged.

The only workaround as of now available to us is to bounce the application server to bring up the ports in listening status.

FYI, the ip's Y.Y.Y.Y and Z.Z.Z.Z can be connected through telnet.

Please suggest how to release these established ports without restarting the application.

Let me know if you require any further information.

Thank U in advance.

To keep the forums high quality for all users, please take the time to format your posts correctly.

First of all, use Code Tags when you post any code or data samples so others can easily read your code. You can easily do this by highlighting your code and then clicking on the # in the editing menu. (You can also type code tags

```text
 and 
```

by hand.)

Second, avoid adding color or different fonts and font size to your posts. Selective use of color to highlight a single word or phrase can be useful at times, but using color, in general, makes the forums harder to read, especially bright colors like red.

Third, be careful when you cut-and-paste, edit any odd characters and make sure all links are working property.

Thank You.

The UNIX and Linux Forums

******************************************************

Are your AIX and WebSphere up-to-date? Do the application logs show something interessting when the problem shows up? Can you and users still open new connections after those 300 are reached via telnet/ssh?

Thanks zaxxon for the reply.

Will keep in mind regarding the the format while posting a thread.

--------------------------------------

  1. 'AIX and WebSphere up-to-date' by any chance is this concerned with the JDK version?

  2. Found the following errors in the application logs while the port is in hanged status:-

 
Exception in Constructor 
sendData:my Data Length is:194
sendData:sending length of:194
 length of the string 194
Exception in sendData64
java.net.ConnectException: Connection timed out: connect
 at java.net.PlainSocketImpl.socketConnect(Native Method)
 at java.net.PlainSocketImpl.doConnect(Unknown Source)
 at java.net.PlainSocketImpl.connectToAddress(Unknown Source)
 at java.net.PlainSocketImpl.connect(Unknown Source)
 at java.net.SocksSocketImpl.connect(Unknown Source)
 at java.net.Socket.connect(Unknown Source)
 at java.net.Socket.connect(Unknown Source)
 at java.net.Socket.<init>(Unknown Source)
 at java.net.Socket.<init>(Unknown Source)

At this point of time when we are trying to connect to the same port it is throwing an error ' Not able to connect to the port'.

  1. Nope, we are not able to create any further connection after the port gets hanged. The output of netstat -an | grep <port no.> shows all the connections in ESTABLISHED status.

Port limit of 300 - Are you sure this isn't a setting in Weblogic, which is perhaps "per applicaiton"?

How to release ports - if they are established, then neither end has closed them. Sounds like the application (client or server end) needs to be modified to close ports when they are no longer needed.

We have confirmed the same from the development team that there is no such setting for the max. no of connections allowed to the ports. Moreover from our analysis point of view the no. 300 may not be hard coded because everytime the no. of established connection varies before the ports get into hanged status, but everytime it is > 300.

Few quick questions for better understanding of the issue:-

1.Is there any other workaround available to analyze from OS point of view before throwing the ball to the application developers court?

2.Please let me know what may be the possible reason or the scenarios where ports remain in ESTABLISHED status.

  1. Is there any system timeout variable/parameter from AIX point of view where we can mention the time after which the established connection will get automatically closed.

  2. Does established connection for a long time mean an IDLE conection?

  3. Is there any possibility that due to some never ending processes these connections are getting created with time?

Being very much new to the platform, asked you all the above questions for better understanding of the issue. Thanks in advance.

From OS view I know no limit how many connections may be opened up to a port. The will be one I think, but we have no problem with applications where more than 300 people connect to a port and there was nothing to be done in the OS to support this.

I did a simple

no -L| grep 300

to see if there is any network option that might have a value of 300 as default and could be taken into relationship with your problem, but I did not find that value.
So I either still think it is an application set parameter or you might have different values on your OS regarding number of connections. Have no better idea, sorry.

Below is the output of the 'no -a' command. Please suggest if any of the parameters in the list may be the root cause of the issue:-

# no -a
arpqsize = 12
arpt_killc = 20
arptab_bsiz = 7
arptab_nb = 149
bcastping = 0
clean_partial_conns = 0
delayack = 0
delayackports = {}
dgd_packets_lost = 3
dgd_ping_time = 5
dgd_retry_time = 5
directed_broadcast = 0
extendednetstats = 0
fasttimo = 200
icmp6_errmsg_rate = 10
icmpaddressmask = 0
ie5_old_multicast_mapping = 0
ifsize = 256
inet_stack_size = 16
ip6_defttl = 64
ip6_prune = 1
ip6forwarding = 0
ip6srcrouteforward = 1
ip_ifdelete_notify = 1
ip_nfrag = 200
ipforwarding = 0
ipfragttl = 2
ipignoreredirects = 0
ipqmaxlen = 100
ipsendredirects = 1
ipsrcrouteforward = 1
ipsrcrouterecv = 0
ipsrcroutesend = 1
llsleep_timeout = 3
lo_perf = 1
lowthresh = 90
main_if6 = 0
main_site6 = 0
maxnip6q = 20
maxttl = 255
medthresh = 95
mpr_policy = 1
multi_homed = 1
nbc_limit = 6935347
nbc_max_cache = 131072
nbc_min_cache = 1
nbc_ofile_hashsz = 12841
nbc_pseg = 0
nbc_pseg_limit = 8159232
ndd_event_name = {all}
ndd_event_tracing = 0
ndp_mmaxtries = 3
ndp_umaxtries = 3
ndpqsize = 50
ndpt_down = 3
ndpt_keep = 120
ndpt_probe = 5
ndpt_reachable = 30
ndpt_retrans = 1
net_buf_size = {all}
net_buf_type = {all}
net_malloc_police = 0
nonlocsrcroute = 0
nstrpush = 8
passive_dgd = 0
pmtu_default_age = 10
pmtu_expire = 10
pmtu_rediscover_interval = 30
psebufcalls = 20
psecache = 1
pseintrstack = 24576
psetimers = 20
rfc1122addrchk = 0
rfc1323 = 0
rfc2414 = 1
route_expire = 1
routerevalidate = 0
rto_high = 64
rto_length = 13
rto_limit = 7
rto_low = 1
sack = 0
sb_max = 1048576
send_file_duration = 300
site6_index = 0
sockthresh = 85
sodebug = 0
sodebug_env = 0
somaxconn = 1024
strctlsz = 1024
strmsgsz = 0
strthresh = 85
strturncnt = 15
subnetsarelocal = 1
tcp_bad_port_limit = 0
tcp_ecn = 0
tcp_ephemeral_high = 65535
tcp_ephemeral_low = 32768
tcp_finwait2 = 1200
tcp_icmpsecure = 0
tcp_init_window = 0
tcp_inpcb_hashtab_siz = 24499
tcp_keepcnt = 8
tcp_keepidle = 14400
tcp_keepinit = 150
tcp_keepintvl = 150
tcp_limited_transmit = 1
tcp_low_rto = 0
tcp_maxburst = 0
tcp_mssdflt = 1460
tcp_nagle_limit = 65535
tcp_nagleoverride = 0
tcp_ndebug = 100
tcp_newreno = 1
tcp_nodelayack = 0
tcp_pmtu_discover = 1
tcp_recvspace = 16384
tcp_sendspace = 16384
tcp_tcpsecure = 0
tcp_timewait = 1
tcp_ttl = 60
tcprexmtthresh = 3
thewall = 8159232
timer_wheel_tick = 0
udp_bad_port_limit = 0
udp_ephemeral_high = 65535
udp_ephemeral_low = 32768
udp_inpcb_hashtab_siz = 24499
udp_pmtu_discover = 1
udp_recvspace = 42080
udp_sendspace = 9216
udp_ttl = 30
udpcksum = 1
use_isno = 1
use_sndbufpool = 1

There is definitely no limit of connections to any particular port in AIX ... I have loads of boxes with 2500+ connections to the port of the oracle listener.

I assume you are using connection pools to access somehow a database. If this is the case, please have a look here Configuring and Using WebLogic JDBC

When your applications attempt to get a connection from a connection pool in which there are no available connections, the connection pool throws an exception stating that a connection is not available in the connection pool. To avoid this error, make sure your connection pool can expand to the size required to accommodate your peak load of connection requests.

To set the maximum number of connections for a connection pool in the Administration Console, expand the navigation tree in the left pane to show the Services�>JDBC�>Connection Pools nodes and select a connection pool. Then, in the right pane, select the Configuration�>Connections tab and specify a value for Maximum Capacity.

Kind regards
zxmaus