Some trace file to track timings of a process

UNIX for Beginners Q & A
1

I will initiate a process from Server1 and the flow is as follow

Server1 --> Web server --> Application server --> DB Server

Note all seperate unix servers.

Now I need to put a trace for that process to track the timings from each server.

Like the below trace should be there:

Server1(09:00:00) --> Web Server(09:00:03) --> Application Server (9:00:05) -->DB Server (9:00:59)
Server1(09:02:30) <-- Web Server(09:02:03) <-- Application Server (9:01:05) <-- DB Server (9:01:03)

With this I can understand in performance bottleneck layer.

Please help.

2

Not sure I understand your request. Pls. enlighten me. What is that flow across four servers? Ususally we're talking of data and / or RPCs. Some processing on server1, then wait for some data transfer or a service call to the Web server, same here and further down, then come back to the parent server1 to continue processing.
Please elaborate.

3

Applications and Performance Management (APM) products may be what you are looking for. Some examples would be nagios, Foglight, Big Brother, Solarwinds, New Relic, BMC, and probably a bunch more. An agent or instrumentation is installed on each system to "monitor" activity. End to end process tracing or "transaction" tracing has been the holy grail of APM for decades and very few companies can do it well because there isn't always an easy mapping between systems. Usually you will get partial info for specific transactions or you can only monitor a few transactions in real time due to the huge volume of data that would need to be collected for all transactions.

4

To get started, have you considered using the web dev water fall performance testing tools in your browser?

That is what I use.

Also, some people find online tools like this helpful:

https://www.webpagetest.org/

These kinds of tests are really good for your first order performance reviews and will tell you where your web performance bottlenecks are including the backend timing.

I use the web dev tools in the chrome browser and I can easily tell what his going on, end-to-end, in a web transaction.

5

Here from server1, I will issue URL link which will go to web server, app server, db server to retrieve the requested data as output

6

This is quite standard, obviously.

This server at unix.com works exactly the same way, as do most web servers.

You should start with a good web dev tool like the one in your browser which is designed to show you the timing in a water fall manner.

You can easily insert some testing code, like some Javascript, in the mix to help you drill down further.

If your backend server is slow, you need to move to the server and look at the system performance as a whole.

If you have any modern day, decent, back-end server, the bottleneck is often in the page rendering on the web site (too many big files, large javascript libs, etc.)

There is not one single tool which will do the entire process from user at the browser to the database.

You must first isolate the problem, normally with a web dev tool in a browser, and then drill down.