NTPSTAT shows clinet is Unsynchronised

I am facing an issue of large offset coming from my GPS clock.
I am syncing RHEL 6.3 server by GPS clock through ntpd service. I have too servers of same type and both are syncing independently to the GPS Clock.

> ntpstat of server 1 gives,

Synchronised to unspecified at stratum 15
time correct to within 145 ms.

> ntpstat of server 2 gives,

Unsynchronised
polling server every 16 s

> ntpq -p of server 1 gives

Reach                Delay             offset                 Jitter
377                    2.644             14654.6            2060.24

> ntpq -p of server 2 gives,

Reach                Delay             offset                 Jitter
377                    2.424             9732.48            3131.24

Ping -c 25 server1 gives,

25 packets transmitted, 25 received, 0% loss, time 24026 ms
rtt min/avg/max/mdev = 1.356/1.860/2.530/0.325 ms

Ping -c 25 server2 gives,

25 packets transmitted, 25 received, 0% loss, time 24002 ms
rtt min/avg/max/mdev = 1.428/1.969/2.525/0.325 ms

Checked and found that ntp.conf file for identical for both the server.

Drift for server 1 is,

428.51

Drift for server 2 is,

456.592

After 15 mins when i check the same parameter, I found the following.
> ntpq -p of server 1 gives

Reach                Delay             offset                 Jitter
377                    2.917            30381.8              3489.88

> ntpq -p of server 2 gives,

Reach                Delay             offset                 Jitter
377                    2.729             28423.7            1746.90

> ntpstat of server 1 gives,

Synchronised to unspecified at stratum 15
time correct to within 165 ms.

> ntpstat of server 2 gives,

Unsynchronised
polling server every 16 s

I also found that there is a difference of 50 seconds between Linux system clock and GPS time in server 1 and 2.
I have run ntpdate -d (GPS Clock IP) and stop the NTPD service.
Run,
> hwclock -S in RHEL system and restarted NTPD service hoping that it will decrease the time between source and synch and NTPD will be able to adjust the time now.But unfortunately offset and jitter in increasing and RHEL servers' time lag increases w.r.t time.

I am finding difficult to understand the above happenings.Can somebody suggest how to troubleshoot the issue?

You should always try to use at least three sources for NTP. The NTP daemon running on your system will throw away the time that has the highest drift and then average the remaining times to set the local system time. This will eliminate any one source that has drifted and reduce the risk of a slow network response.

In short, try using three sources and see if that fixes your problem.

OK.But what is the primary reason for this?Can it be identified from the input that I have provided.

It looks like there is something wrong with the time that is being served by server2 and it's configuration. This would be solved by having 3 credible sources.

If ntpq -p shows high and growing offsets, then usually it means that the clock control in the kernel does not work.
So it looks that both server1 and server2 are suffering.
But I wonder why ntpstat on server1 looks okay.?
Are server1 and server2 virtual? I have seen this on VMware ESX guests: the clock steppers are ignored, and the clock needs to be set on the VMware ESX host.

No.It is not VM. The machines are physical machine where RHEL 5.9 is installed.

After a lot of brainstorming and consulting with some NTP expert I found the following.

 2 May 17:00:28 ntpd[2178]: synchronized to 172.31.64.10, stratum 1
 2 May 17:36:39 ntpd[2178]: time reset -0.313640 s
 2 May 17:37:13 ntpd[2178]: synchronized to 172.31.64.10, stratum 1
 2 May 17:52:12 ntpd[2178]: time reset +0.491725 s
 2 May 17:52:31 ntpd[2178]: synchronized to 172.31.64.10, stratum 1
 2 May 18:08:34 ntpd[2178]: time reset +0.283749 s
 2 May 18:09:49 ntpd[2178]: synchronized to 172.31.64.10, stratum 1
 2 May 18:50:31 ntpd[2178]: time reset -0.312960 s
 2 May 18:51:34 ntpd[2178]: synchronized to 172.31.64.10, stratum 1
 2 May 19:06:30 ntpd[2178]: time reset +0.481487 s

NTP expert comment:

This means whenever the system time is synchronized, a few minutes late
the time offset is again so large (> 128 ms) that ntpd is unable to
correct it smoothly. So it steps the system time ("time reset") and
restarts from scratch.

So this looks like the timekeeping in your Linux system is broken, i.e.
the system time increases not continuously at the same rate.

NTP is unable to fix some broken timekeeping. Only if the timekeeping is
stable, NTP can determine the current offset, the time drift, and adjust
the system time smoothly so that the time offset is continuously as
small as possible.

I don't thing there's a problem with the package exchange. It's just the
Linux system time which has too much jitter, i.e. changes quickly from
increasing too faster to increasing too slow. You need to fix that
first, otherwise you are out of luck.

Now the question is How to fix the "BROKEN TIMEKEEPING OF LINUX System"?
Can some body help me out?

Wrong tags. [code], not <code> . Thanks for trying this time though. Next time I suggest the preview button, so you can tell that your post is coming out flagrantly wrong.

Well, would need to find out why timekeeping is broken. It's likely a hardware issue. What's your system?

My system is based on RHEL 6.2.I can tell you the server version tomorrow.

Does

hwclock

return a plausible time?

The hardware would be more relevant. Also look for any interesting-looking messages in dmesg | less like "losing some ticks".

Run this command.

dmesg | less

.
But did not find any message like

 losing some tics 

[/FONT][/COLOR]

---------- Post updated at 05:12 AM ---------- Previous update was at 04:37 AM ----------

Give Date command in the server.It gives

 Fri 12 May 2017 15:38:53 IST 2017

At the same time give

hwclock

It gives,

 Fri 12 May 2017 03:38:58 PM IST -0.596935