I have one application server, which is Solaris non global zone (dsapid02-dt3d02 - 10.63.76.96) and a database server, which is Linux (db1-serv26 - 10.63.36.117 & 10.63.36.118).
Whenever database admin is starting listener, it starts filling listener log file. Within minutes, it will make file in gigabytes.
Scenerio - There were 5 databases running on db1-serv26. One of the database, T3IA was using port 1558. A month back all those 5 databases were migrated to another server and all file-systems on db1-serv26 were removed after that, so no listener was listening on 1558 and other ports used by T3 QA databases on db1-serv26.
Now there is a new requirement of creation of new database on db1-serv26, which should use port 1558. The moment database administrator configure 1558 port, a number of processes from IP 10.63.76.96 (dsapid02-dt3d02) from different ports (59881, 59882, 59883, 59884, 59885, 59886, etc) started trying to connect to T3IA database on db1-serv26 on port 1558. As 1558 is now being used by newely created database, and not by T3IA, the connections were not successful, and hence started generating following errors in large number
TNS-12514: TNS:listener does not currently know of service requested in connect descriptor
18-OCT-2013 22:32:39 * (CONNECT_DATA=(CID=(PROGRAM=)(HOST=__jdbc__)(USER=))(SERVICE_NAME=T3IA.tms.toyota.com)) * (ADDRESS=(PROTOCOL=tcp)(HOST=10.63.76.96)(PORT=39372)) * establish * T3IA.tms.toyota.com * 12514
TNS-12514: TNS:listener does not currently know of service requested in connect descriptor
18-OCT-2013 22:32:39 * (CONNECT_DATA=(CID=(PROGRAM=)(HOST=__jdbc__)(USER=))(SERVICE_NAME=T3IA.tms.toyota.com)) * (ADDRESS=(PROTOCOL=tcp)(HOST=10.63.76.96)(PORT=39385)) * establish * T3IA.tms.toyota.com * 12514
If I check 1558 port is being used by which process (through a script), I get this output.
root@dsapid02-dt3d02:/# /tmp/check_port
Enter port you would like to know pid for: 1558
---------------------------------------------
Port: 1558 is being used by PID: 1330 java T3LinkValidator
---------------------------------------------
Port: 1558 is being used by PID: 5569 java T3LinkValidator
---------------------------------------------
Port: 1558 is being used by PID:23183 java T3LinkValidator
---------------------------------------------
Port: 1558 is being used by PID: 6411 java T3LinkValidator
---------------------------------------------
Port: 1558 is being used by PID:17983 java T3LinkValidator
---------------------------------------------
Port: 1558 is being used by PID: 2958 java T3LinkValidator
---------------------------------------------
Port: 1558 is being used by PID:28607 java T3LinkValidator
---------------------------------------------
Port: 1558 is being used by PID:21057 java T3LinkValidator
---------------------------------------------
Port: 1558 is being used by PID:17826 java T3LinkValidator
---------------------------------------------
Port: 1558 is being used by PID:29999 java T3LinkValidator
---------------------------------------------
Port: 1558 is being used by PID: 1888 java T3LinkValidator
---------------------------------------------
Port: 1558 is being used by PID:11236 /t3/apps/Oracle/Middleware/10.3.5/jdk160_24/bin/java -server -Xms2048m -Xmx2048
---------------------------------------------
Port: 1558 is being used by PID:13689 java T3LinkValidator
---------------------------------------------
Port: 1558 is being used by PID:20985 java T3LinkValidator
---------------------------------------------
Port: 1558 is being used by PID: 1672 java T3LinkValidator
---------------------------------------------
Port: 1558 is being used by PID:23303 java T3LinkValidator
^Croot@dsapid02-dt3d02:/# ps -ef | grep T3LinkValidator | wc -l
22
root@dsapid02-dt3d02:/#
Are you asking me this one ?
Yes, I can run "netstat -p" on my server.
What's that T3LinkValidator for? Checking an invalid link (as you moved the database to another server)? Can you point it to the correct server? Can you stop/remove it, as it has proven to be unnecessary for a month?
Is it something, which needs to be taken care by owner of this application to point to correct db server and not to be taken care by db admin or sys admin ? If I know, it is broken at which level, I can get hold of that person/team.
The owner is the right person to talk to. That thing is obiously broken as it it validating an inexistent link. If that didn't catch anybody's eye, chances are good that it can go away without replacement.