Cant find Server connections for a web logic Managed Server on Port 80

Good morning, I need your help Please, In a Production Web application System Im trying to get Server connections for a web logic Managed Server

This is the Scenario:

Client - Weblogic - Server connections:

On Client Side Ive got These Connections on Port 80, so its OK:

promrecharges01.inet:/home/perm_admin> lsof -i tcp:80

COMMAND   PID       USER   FD   TYPE     DEVICE SIZE/OFF NODE NAME

java    16801 perm_admin  112u  IPv6 1275987347      0t0  TCP promrecharges01.inet:58303->lbapp1.nh.inet:http (ESTABLISHED)

java    16801 perm_admin  118u  IPv6 1275987348      0t0  TCP promrecharges01.inet:58306->lbapp1.nh.inet:http (ESTABLISHED)

java    16801 perm_admin  150u  IPv6 1275989478      0t0  TCP promrecharges01.inet:38944->10.80.6.237:http (ESTABLISHED)

java    16801 perm_admin  159u  IPv6 1251363681      0t0  TCP promrecharges01.inet:58571->lbapp1.nh.inet:http (ESTABLISHED)

This is the same without DNS:

java      16801 perm_admin   97u  IPv6 1276012449      0t0  TCP 10.203.216.10:44779->10.203.4.34:80 (ESTABLISHED)                                                                              

java      16801 perm_admin  109u  IPv6 1276010214      0t0  TCP 10.203.216.10:44298->10.203.4.34:80 (ESTABLISHED)                                                                              

java      16801 perm_admin  133u  IPv6 1276010279      0t0  TCP 10.203.216.10:53812->10.80.6.237:80 (ESTABLISHED)                                                                              

java      16801 perm_admin  157u  IPv6 1276010443      0t0  TCP 10.203.216.10:44800->10.203.4.34:80 (ESTABLISHED)                                                                               

java      16801 perm_admin  159u  IPv6 1251363681      0t0  TCP 10.203.216.10:58571->10.203.4.34:80 (ESTABLISHED)   

The application is a websocket:

Ip 10.203.4.34 is the Load balancer and port 80 consume a web service
I can reach out 10.203.4.34 and port 80 using telnet to test connectivity

From THe SErver Side is where Ive got the issue:
Its a Managed Server weblogic:

The owner of Middleware processes are offerings:

offerings   10391 56315  0 16:23 ?        00:00:00 /usr/sbin/sendmail -FCronDaemon -i -odi -oem -oi -t -f root

offerings   10400 10391  0 16:23 ?        00:00:00 /usr/sbin/postdrop -r

offerings   28628 38395  0 Mar06 ?        00:00:00 /bin/sh /u01/domains/coWeblogic/bin/startWebLogic.sh

offerings   28685 28628 47 Mar06 ?        19:29:59 /u01/jdk/bin/java -server -Xms6144m -Xmx6144m -cp /u01/producto/wlserver/server/lib/weblogic-launcher.jar -Dlaunch.use.env.classpath=true -Dwebl

offerings   38347     1  0 Mar06 ?        00:00:00 /bin/sh ./startNodeManager.sh

But I cant find connections on Port 80:

promthapp02c:/home/ERM_Reca> lsof -i tcp:80
promthapp02c:/home/ERM_Reca>

I dont know if it has something to do whith the user I own (ERM_Reca) instead offerings owner?
Im not the owner of this user (offerings) but The Middleware guys so obviusly I have not permissions
The purpose for this is to find and connectivity issue fom the Server (middleware side)

SEcondly Id like to know what is the meaning of these connections in a foreign adress column?
What is fff:10.203.5.254 ? I cant find the server name for fff:10.203.5.254? its a kind of broadcast?

tcp        0      0 10.203.5.22:52725           10.203.5.206:4100           ESTABLISHED
tcp        0      0 :::4100                     :::*                        LISTEN     
tcp        0      0 ::ffff:10.203.5.22:4100     ::ffff:10.203.5.254:44773   ESTABLISHED
tcp        0      0 ::ffff:10.203.5.22:4100     ::ffff:10.203.5.254:65192   ESTABLISHED
tcp        0      0 ::ffff:10.203.5.22:4100     ::ffff:10.203.5.254:22465   ESTABLISHED
tcp        0      0 ::ffff:10.203.5.22:4100     ::ffff:10.80.1.253:11304    ESTABLISHED

tcp        0      0 ::ffff:10.203.5.22:4100     ::ffff:10.203.5.254:20157   ESTABLISHED

I appreciate your help in advanced

Hello,

If I understand you correctly, you have the following kind of setup:

• Clients send Web requests to a load balancer
• Load balancer is configured to take those requests on a load-balanced virtual IP on port 80, and then funnel them to the back-end application servers
• The application servers accept the connections from the load balancer on their application port, and return a response
• The load balancer returns the response that it obtained from the application servers to the clients, via the established HTTP session to the load-balanced VIP

And specifically, it is this last part of the process - the connection from the load balancer to the back-end servers - that is not working. Is that correct ?

If so, then you'd be best to check the load balancer logs in the first instance. Almost all load balancer software or load balancing appliances will log both the connections they receive, and the current state of the back-end servers which are in the load-balanced pool. The load balancer logs should reveal whether it regards the back-end servers as being up or not.

It's also entirely possible that the back-end servers are running their applications on a port other than 80. The load balancer, for instance, could take HTTP connections on port 80, and then pass those through to the back-end servers on a totally different port such as 8080 or 9090. The back-end servers would then return their response, and the clients would appear to be getting a normal HTTP Web response from the load-balanced VIP.

So you might not be seeing port 80 listening on the back-end servers either because their Web server is down (which would explain why things aren't working), or because the back-end servers do not in fact listen on port 80 for connections from the load balancer.

Either way, your best bet is to check out the load balancer, examining its logs and its current configuration to determine the port it tries to talk to the back-end servers on, and then take things from there.

Also, regarding your second question: what this output shows is a number of outbound connections from whatever runs on the remote IP 10.203.5.254, to the TCP port 4100 on the local IP 10.203.5.22. Beyond that, only you (or someone at your local site who is more familiar with the network, server or application setup) would be able to tell you what these IPs are, or what they represent.

Thanks you very much for youre time and support, it is useful to me your update
Actually Ive got not Issue with the load balancer i think, its just a a way to go deeper and Understand conectivity back and forth among client(OK) load balancer- backend server(web logic No)

bcz I can see client connections on port 80 but On the backend Server side I cant, it does not make sense

The Othe Hypothesis:
It's also entirely possible that the back-end servers are running their applications on a port other than 80
How do i figure out?

Only I now from the backend it has connections to ports 7003 and 7005(weblogic), and uses load balancer ip address and a webservice

http://10.203.4.34:7003/MTHService/RemoreRequest?WSDL

I appreciate your help in advanced

What is the nature of the load balancer ? Is it a dedicated hardware appliance, or a server ? If it's a server, what operating system and load balancing software is it running ? In order to understand what the load balancer is doing, you'd first have to know what it actually is, and then from there, you can examine its configuration to see how it's set up. How you'll do that depends entirely on the nature of the load balancer itself.

Thanks you very much for your useful insight
Understood your suggestion, Im not in on the Network Team but I asked one of them and He said the Load Balancer is an appliance and it is an A10 using round robbin
I know there are tons info about Load Balancers but do you know any special link about the nitty griity for load balancers?
I appreciate your help in advanced

Hello,

I'm not familiar with A10, so I'm afraid I can't really tell you anything specific about either the company or their products. The best people to talk to again would be your network team who manage this A10 load balancer solution. They should be able to confirm for you which back-end servers are in the load balancer pool; what IP address and on which TCP port the load balancer talks to them on; and how the status of the back-end servers is checked from the load balancer's perspective (e.g. a simple TCP port connection check; a request for a specific URL and a search for a given string within the content returned from that URL; etc.).

In terms of general information about A10's load balancing solutions, they seem to have a fairly comprehensive Web page, which you can find here:

What is a Load Balancer and How Does Load Balancing Work ?

But again, when it comes to understanding how your own network is set up, there's no substitute for local knowledge. Asking someone from your network team to talk you through how the whole solution is set up for your particular environment would be the best way to proceed, I would imagine. It would probably also be a good idea for them to document this somewhere, if they haven't already done so.

Hope this helps !

The ::ffff: prefix is expained here:

First, i would use lsof with root user (or sudo)
Information you get from user-invoked lsof is limited.

As for LB problematics, what kind of LB this is, IP layer (layer 4) or layer 7 (usually http application layer) ?
Is layer 2 involved (MAC layer) e.g do you have loopback address (VIPs) on java servers as well or ?
This is important info, to determine if and how traffic is routed to applications.

In any case, a tcpdump would help you clarify which backends the LB is hitting under what conditions IF not routed internaly by java application server by URL or likes (layer 7).

Java applications servers in general have multiple apps on different ports (say 8080, 8081 etc.), even have builtin proxies in front - which serve as url rewriters/routers to appropriate java applications.

To put it in example, you can have 2 urls

http://my-wonder.com/something
http://my-wonder.com/otherthing

Which hits same backend IP(s) over VIP address, but different applications and ports e.g /something is served by java application running on port 8080 on backend server A, and /otherthing is served by java application running port 8081 on same backend server.

This is simple example, things get quite complicated in jboss/websphere/weblogic world quite quick, as you most often have cluster (multiples of jboss/websphere/weblogic) and those have proxies inside them self and manipulate content on layer 7.

So one should look at LB config, followed by configuration of java application server, and applications hosted on them to see the general picture where and what client traffic hits.

There is no magic command which will analyse these things on all layers mentioned above, only by observing each component and how it interacts with others one can draw conclusions.
Or by reading documentation which hopefully exists for such systems

Regards
Peasant.

Thanks you very much all of you for your time, effort and useful insights regarding load balancers, ip addresses explanation and documentation
The ip layer I think is 7 http application layer
Now it is up to me to ask the network Team to help me out with some LB configuration or LB logs, what Im not sure if they provide me such that info bcz of Security Reasons or something like that or if there is not an ongoing incident
Thanks you very much all of you

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.