How to check, what NTP clients are synching time with NTP servers?

solaris_1977 · October 25, 2021, 4:14pm

Hi,
I have an old Solaris 10 sparc server, which is working as NTP server. It is synching its time from GPS clocks. I want to decomm this server and move NTP service to new Linux machine. That will include hostname/IP change, which means I will need to update ntp.conf of every client which are synching time with this Solaris 10 NTP server. Is there any easy way to know, what all servers are contacting this NTP server for time synch ? If I know all clients I can update all of them via ansible.
Theoretically one way to capture output of netstat to a file and sort/grep for unique IP/hostname of clients and let it run for 2 days. But I wouldn't know when NTP sync will happen and at that time it would be capturing output.
Please advise.
Thanks

drysdalk · October 25, 2021, 5:58pm

Hello,

One option here might be to sniff the traffic with the snoop command. I've not got a Solaris box immediately to hand to test with, but off the top off my head something like snoop -V port ntp might do the trick. The idea would be to record the traffic between the server and its clients, and see which clients are connecting that way. You can also just leave snoop capturing traffic and writing the data to a file, and then examine the data later with the snoop command at your leisure.

Alternatively, you could set up ipfilter rules to log all traffic on port 123, and examine the firewall logs to see the clients connecting. Lastly, you may of course be able to configure ntpd itself to run in debug mode, but you'll get an excessive amount of log data that way, and this may not be an option depending on the version of NTP that you're running.

Anyway, hope one of these suggestions helps ! If they won't work for you or if you have any other questions, please let us know and we can take things from there.

hicksd8 · October 25, 2021, 6:21pm

I think you might do what you want with monlist

See this thread from our legacy site:

I think that you will only get the last 600 NTP requests listed though. How many clients do you think there are?

Search the web for monlist anyway.

solaris_1977 · October 25, 2021, 6:33pm

This server doesn't have ntpdc. I can see only ntpdate, ntpq, ntptrace
There should be around 1000 clients.

If I go with snoop solution, then I get this -

ntp-master1-prod # snoop -V port ntp
Using device /dev/bge0 (promiscuous mode)
________________________________
ng-i-adm1-admin.dupedro.com -> ntp-master1-prod.dupedro.com ETHER Type=0800 (IP), size = 90 bytes
ng-i-adm1-admin.dupedro.com -> ntp-master1-prod.dupedro.com IP  D=192.168.243.10 S=192.168.245.71 LEN=76, ID=10833, TOS=0x0, TTL=253
ng-i-adm1-admin.dupedro.com -> ntp-master1-prod.dupedro.com UDP D=123 S=123 LEN=56
ng-i-adm1-admin.dupedro.com -> ntp-master1-prod.dupedro.com NTP  client [st=4] (2021-10-25 11:11:59.49193)
________________________________
ntp-master1-prod.dupedro.com -> ng-i-adm1-admin.dupedro.com ETHER Type=0800 (IP), size = 90 bytes
ntp-master1-prod.dupedro.com -> ng-i-adm1-admin.dupedro.com IP  D=192.168.245.71 S=192.168.243.10 LEN=76, ID=56797, TOS=0x0, TTL=255
ntp-master1-prod.dupedro.com -> ng-i-adm1-admin.dupedro.com UDP D=123 S=123 LEN=56
ntp-master1-prod.dupedro.com -> ng-i-adm1-admin.dupedro.com NTP  server [st=3] (2021-10-25 11:11:59.48939)
________________________________
sbsdi-mysqldb3-prod.dupedro.com -> ntp-master1-prod.dupedro.com ETHER Type=0800 (IP), size = 90 bytes
sbsdi-mysqldb3-prod.dupedro.com -> ntp-master1-prod.dupedro.com IP  D=192.168.243.10 S=192.168.242.239 LEN=76, ID=20125, TOS=0x0, TTL=254
sbsdi-mysqldb3-prod.dupedro.com -> ntp-master1-prod.dupedro.com UDP D=123 S=123 LEN=56
sbsdi-mysqldb3-prod.dupedro.com -> ntp-master1-prod.dupedro.com NTP  client [st=4] (2021-10-25 11:12:07.51855)
________________________________
ntp-master1-prod.dupedro.com -> sbsdi-mysqldb3-prod.dupedro.com ETHER Type=0800 (IP), size = 90 bytes
ntp-master1-prod.dupedro.com -> sbsdi-mysqldb3-prod.dupedro.com IP  D=192.168.242.239 S=192.168.243.10 LEN=76, ID=64827, TOS=0x0, TTL=255
ntp-master1-prod.dupedro.com -> sbsdi-mysqldb3-prod.dupedro.com UDP D=123 S=123 LEN=56
ntp-master1-prod.dupedro.com -> sbsdi-mysqldb3-prod.dupedro.com NTP  server [st=3] (2021-10-25 11:12:07.51892)
________________________________
iotwa-dell-swa1.dupedro.com -> ntp-master1-prod.dupedro.com ETHER Type=0800 (IP), size = 90 bytes
iotwa-dell-swa1.dupedro.com -> ntp-master1-prod.dupedro.com IP  D=192.168.243.10 S=192.168.21.217 LEN=76, ID=55037, TOS=0x0, TTL=62
iotwa-dell-swa1.dupedro.com -> ntp-master1-prod.dupedro.com UDP D=123 S=123 LEN=56
iotwa-dell-swa1.dupedro.com -> ntp-master1-prod.dupedro.com NTP  client [st=4] (2021-10-25 11:12:12.73756)
________________________________
ntp-master1-prod.dupedro.com -> iotwa-dell-swa1.dupedro.com ETHER Type=0800 (IP), size = 90 bytes
ntp-master1-prod.dupedro.com -> iotwa-dell-swa1.dupedro.com IP  D=192.168.21.217 S=192.168.243.10 LEN=76, ID=33393, TOS=0x0, TTL=255
ntp-master1-prod.dupedro.com -> iotwa-dell-swa1.dupedro.com UDP D=123 S=123 LEN=56
ntp-master1-prod.dupedro.com -> iotwa-dell-swa1.dupedro.com NTP  server [st=3] (2021-10-25 11:12:12.74035)
^Cntp-master1-prod #

If there is no straightforward solution, then I can redirect this output to a file and later I can do something like this, should it give me what I want? -

cat /var/tmp/123-port.out | grep server | awk '{print $3}'

And then do uniq.

How frequent NTP synchs between client-server? Should I let it run for a day or two or so or few hours should work?

munkeHoller · October 25, 2021, 7:00pm

some observations

I see both 'client' and 'server' with NTP entries.
List item "$3" is what exactly ?, if you provide an example of what you need/expect folks will be better equipped to provide a concise answer. for example if you want lines with NTP and server ... something along the following awk '/NTP.*server*/{print}' /var/tmp/123-port.out will get you what is needed, furthermore awk can build you the uniq set of occurances, if you give an example of what you are expecting ....
no need to cat the file, awk can read that directly just add to the end of the command awk '{print $3}' /var/tmp/123-port.out

solaris_1977 · October 25, 2021, 9:06pm

This NTP server depend on other 2 Master NTP servers and those 2 connects to GPS clocks. This is how it is setup.

ntp-master1-prod # ntpq -p
     remote           refid      st t when poll reach   delay   offset    disp
==============================================================================
+wksa-app1-prod. sea-gps-clock2.  2 u    -  512  377     0.70   -0.012    0.05
*wksa-app2-prod. sea-gps-clock1.  2 u  215  512  377     0.66   -0.026    0.05
ntp-master1-prod #

I need to know, what all clients connects to ntp-master1-prod, so I can edit ntp.conf on all of them to update.

With '{print $3}' I was trying to get the client hostname. Is it correct way?
For how long I can keep it running? Maybe it is dependent on how frequent NTP synchs?

drysdalk · October 25, 2021, 10:46pm

Hello,

Re-directing the output of a snoop running in your terminal to a file, and manually parsing that output, is one way to do it. But a better way would just be to directly run snoop to write its captured packets to a file rather than displaying a summary of their contents on your terminal (the -o flag does this I believe). You could then re-play that packet capture file with snoop -i at your leisure at any time, and could apply the same kinds of filters and parameters (e.g. port ntp) when doing so. That's probably a more efficient way of doing things, overall.

Either way, once you have your output in a human-readable format from either a live packet dump or a replay of a packet capture, you could then strip out the parts you want in the manner you describe. You may also wish to use the -n flag to snoop to just get numeric IPs rather than hostnames, if that helps your parsing at all.

Hope this helps !

munkeHoller · October 25, 2021, 11:11pm

hi, if $3 is the client, then yes. try this script and see if it gives you what you are looking for, it eliminates the need for any uniq processing as that's implicit in the clients[] associative array.
you definitely need to capture connections over multiple days , perhaps covering a system-wide reboot as some clients may not be active (for maintenance etc ).

        clients[$3]++;
}
END{
        for ( client in clients )
                printf( "%s connected %d times to ntp-master1-prod\n", client, clients[client] )
}

solaris_1977 · October 25, 2021, 11:45pm

@drysdalk - snoop is good option, I will try it.

@munkeHoller - How will I run this script. This way, as below ?

ntp-master1-prod # cat /var/tmp/ntp-client-chk1.sh
        clients[$3]++;
}
END{
        for ( client in clients )
                printf( "%s connected %d times to ntp-master1-prod\n", client, clients[client] )
}
ntp-master1-prod # /var/tmp/ntp-client-chk.sh
/var/tmp/ntp-client-chk.sh: line 1: clients[]++: command not found
/var/tmp/ntp-client-chk.sh: line 2: syntax error near unexpected token `}'
/var/tmp/ntp-client-chk.sh: line 2: `}'
ntp-master1-prod #

munkeHoller · October 25, 2021, 11:54pm

hmm, i see my previous post missed some text off, apologies.
save the awk script to a file, lets call it snoop.awk

that should contain:

/ntp-master1-prod.*NTP.*server*/{
	clients[$3]++;
}
END{
	for ( client in clients )
		printf( "%s connected %d times to ntp-master1-prod\n", client, clients[client] )
}

then execute the script as follows:

awk -f snoop.awk file-you-want-to-process

let me know how that goes.

solaris_1977 · October 26, 2021, 12:17am

What should I have in 'file-you-want-to-process' ? Looks like, I am missing something else

ntp-master1-prod # cat snoop.awk
/ntp-master1-prod.*NTP.*server*/{
        clients[$3]++;
}
END{
        for ( client in clients )
                printf( "%s connected %d times to ntp-master1-prod\n", client, clients[client] )
}
ntp-master1-prod #
ntp-master1-prod # awk -f snoop.awk all-ntp-clients
awk: can't open all-ntp-clients
ntp-master1-prod #

munkeHoller · October 26, 2021, 12:44am

the file all-ntp-clients seems to be missing ...
its contents should be similar to what you showed on your posting using snoop

ng-i-adm1-admin.dupedro.com -> ntp-master1-prod.dupedro.com ETHER Type=0800 (IP), size = 90 bytes
ng-i-adm1-admin.dupedro.com -> ntp-master1-prod.dupedro.com IP D=192.168.243.10 S=192.168.245.71 LEN=76, ID=10833, TOS=0x0, TTL=253
ng-i-adm1-admin.dupedro.com -> ntp-master1-prod.dupedro.com UDP D=123 S=123 LEN=56
ng-i-adm1-admin.dupedro.com -> ntp-master1-prod.dupedro.com NTP client [st=4] (2021-10-25 11:11:59.49193)
ntp-master1-prod.dupedro.com -> ng-i-adm1-admin.dupedro.com ETHER Type=0800 (IP), size = 90 bytes
ntp-master1-prod.dupedro.com -> ng-i-adm1-admin.dupedro.com IP D=192.168.245.71 S=192.168.243.10 LEN=76, ID=56797, TOS=0x0, TTL=255
ntp-master1-prod.dupedro.com -> ng-i-adm1-admin.dupedro.com UDP D=123 S=123 LEN=56
ntp-master1-prod.dupedro.com -> ng-i-adm1-admin.dupedro.com NTP server [st=3] (2021-10-25 11:11:59.48939)
sbsdi-mysqldb3-prod.dupedro.com -> ntp-master1-prod.dupedro.com ETHER Type=0800 (IP), size = 90 bytes
sbsdi-mysqldb3-prod.dupedro.com -> ntp-master1-prod.dupedro.com IP D=192.168.243.10 S=192.168.242.239 LEN=76, ID=20125, TOS=0x0, TTL=254
sbsdi-mysqldb3-prod.dupedro.com -> ntp-master1-prod.dupedro.com UDP D=123 S=123 LEN=56
sbsdi-mysqldb3-prod.dupedro.com -> ntp-master1-prod.dupedro.com NTP client [st=4] (2021-10-25 11:12:07.51855)

solaris_1977 · October 26, 2021, 1:10am

Yes, I am in better shape now.

ntp-master1-prod # snoop -V port 123 >> /var/tmp/ntp-stats
Using device /dev/bge0 (promiscuous mode)
^Cntp-master1-prod #
ntp-master1-prod # head /var/tmp/ntp-stats
________________________________
smng-iot-zonemgr1-prod.dupedro.com -> ntp-master1-prod.dupedro.com ETHER Type=0800 (IP), size = 90 bytes
smng-iot-zonemgr1-prod.dupedro.com -> ntp-master1-prod.dupedro.com IP  D=192.168.243.10 S=216.221.133.196 LEN=76, ID=17388, TOS=0x0, TTL=254
smng-iot-zonemgr1-prod.dupedro.com -> ntp-master1-prod.dupedro.com UDP D=123 S=123 LEN=56
smng-iot-zonemgr1-prod.dupedro.com -> ntp-master1-prod.dupedro.com NTP  client [st=4] (2021-10-25 17:49:06.71605)
________________________________
ntp-master1-prod.dupedro.com -> smng-iot-zonemgr1-prod.dupedro.com ETHER Type=0800 (IP), size = 90 bytes
ntp-master1-prod.dupedro.com -> smng-iot-zonemgr1-prod.dupedro.com IP  D=216.221.133.196 S=192.168.243.10 LEN=76, ID=31822, TOS=0x0, TTL=255
ntp-master1-prod.dupedro.com -> smng-iot-zonemgr1-prod.dupedro.com UDP D=123 S=123 LEN=56
ntp-master1-prod.dupedro.com -> smng-iot-zonemgr1-prod.dupedro.com NTP  server [st=3] (2021-10-25 17:49:06.71647)
ntp-master1-prod #
ntp-master1-prod # cat snoop.awk
/ntp-master1-prod.*NTP.*server*/{
        clients[$3]++;
}
END{
        for ( client in clients )
                printf( "%s connected %d times to ntp-master1-prod\n", client, clients[client] )
}
ntp-master1-prod #
ntp-master1-prod # awk -f snoop.awk /var/tmp/ntp-stats
ntp-master1-prod.dupedro.com connected 1 times to ntp-master1-prod
smng-iot-zonemgr1-prod.dupedro.com connected 1 times to ntp-master1-prod
wksa-staging2-prod.dupedro.com connected 1 times to ntp-master1-prod
ng-i-lngapp1-admin.dupedro.com connected 1 times to ntp-master1-prod
e911i-alilink-i-prod.dupedro.com connected 1 times to ntp-master1-prod
sbsdi-services5-prod.dupedro.com connected 1 times to ntp-master1-prod
iowa-pici-imf-1a-prod.dupedro.com connected 2 times to ntp-master1-prod
iotwa-dell-swa1.dupedro.com connected 1 times to ntp-master1-prod
nmsi-rmc1-prod.dupedro.com connected 1 times to ntp-master1-prod
ntp-master1-prod #

Regarding, how long I should keep the snoop running, I am was reading some notes that pools are done at minimum time is 64 sec and maximum time 1024 sec. So I am guessing, couple of hours of run should collect all servers. We don't have any customize setting in ntp.conf, all are by default. All our servers are either Solaris (9, 10, 11) and Linux (RHEL 6/7)

Neo · October 26, 2021, 1:45am

You might also consider modifying the NTPD code and add a few lines for logging connections; and recompiling.

Also, you might also consider running chronyd instead of ntpd. chrony has a much better track record regarding security. Search these forums for a number of my prior posts on this topic.

FWIW, I only run chronyd because it is more secure and less buggy.

solaris_1977 · October 26, 2021, 4:42am

Yes, I agree.
That is the reason I am migrating NTP service to latest OS, on chronyd. Existing one is running on very old hardware (V210).
Thanks

Neo · October 26, 2021, 9:49am

Network history says.... Someone named Neo" advised you to move to chronyd nearly two years ago:

I think you can compile chronyd for Solaris 10.

Did you try?

https://git.tuxfamily.org/chrony/chrony.git/

MadeInGermany · October 26, 2021, 10:53am

Yes, the poll intervals starts with 64 seconds then, if the source is stable, grows to the 1024 seconds maximum.
The current poll interval is shown in ntpq -p

Neo · October 26, 2021, 11:46am

mlichvar/chrony/blob/master/doc/installation.adoc

// This file is part of chrony
//
// Copyright (C) Richard P. Curnow  1997-2003
// Copyright (C) Miroslav Lichvar  2009-2016
//
// This program is free software; you can redistribute it and/or modify
// it under the terms of version 2 of the GNU General Public License as
// published by the Free Software Foundation.
//
// This program is distributed in the hope that it will be useful, but
// WITHOUT ANY WARRANTY; without even the implied warranty of
// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
// General Public License for more details.
//
// You should have received a copy of the GNU General Public License along
// with this program; if not, write to the Free Software Foundation, Inc.,
// 51 Franklin Street, Fifth Floor, Boston, MA  02110-1301, USA.

= Installation

This file has been truncated. show original